+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210507/A0j --output-run DEEP/20210507/A0j/science#step1/20240315T041214Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 1.5719 seconds; current memory usage: 0.179 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.191 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0082 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0217 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/pipeline_orig.qgraph DEEP/20210507/A0j/science#step1/20240315T041214Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.5580 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 15.7832 seconds; current memory usage: 0.381 Gibyte, delta: 0.201 Gibyte, peak delta: 0.190 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.381 Gibyte (main), 0.369 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.2218 seconds; current memory usage: 0.381 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.381 Gibyte (main), 0.369 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 1116 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210507_A0j_science#step1_20240315T041214Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.2180 seconds; current memory usage: 0.381 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.381 Gibyte (main), 0.369 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 1118 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.3594 seconds; current memory usage: 0.382 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.382 Gibyte (main), 0.369 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: e07746cd-8771-443d-8946-a053d93ef20d parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-15 04:13:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-15 04:13:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 04:13:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-15 04:13:01 proc_lsst.multi:170 [INFO] [multi] job_id 55323 proc_lsst.multi INFO: [multi] job_id 55323 2024-03-15 04:13:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:01 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 55323 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 55323 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 55323 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 4b15ab1b57043d7a4cf47052602070ce parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b849a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/e9fc1f26-3afe-4d30-b6fe-dc96ca8b164e_isr_994638_3.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/e9fc1f26-3afe-4d30-b6fe-dc96ca8b164e_isr_994638_3.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 7d03add13cc7892a5fc548dce6590113 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b85260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/01983423-b790-4d50-9042-5f33de67dd99_isr_994765_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/01983423-b790-4d50-9042-5f33de67dd99_isr_994765_53.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash ed98e00f106740672aac88a93e8c56eb parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/f218bee6-7b15-497b-a39a-91adbe3afc0f_isr_994639_37.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/f218bee6-7b15-497b-a39a-91adbe3afc0f_isr_994639_37.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 61b4a360897f01cbaf85c7c7074517cb parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b856c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4d5b323b-e795-40b4-bef9-b82fcecc2288_isr_994766_23.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4d5b323b-e795-40b4-bef9-b82fcecc2288_isr_994766_23.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 0b811b2ed7567efd2039637c49ad4496 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b85bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3a961bae-9655-4fb3-a4e2-907830049297_isr_994638_50.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3a961bae-9655-4fb3-a4e2-907830049297_isr_994638_50.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 942958f49bc53a3a3fecc0484fd14d26 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6c593c56-4817-4ea2-ac03-e91f4a652b16_isr_994638_28.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6c593c56-4817-4ea2-ac03-e91f4a652b16_isr_994638_28.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 872c8bb6ba02c9d2e65b725890f1698f parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d2803a60-65f4-4306-8753-d06a4fee97e0_isr_994640_30.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d2803a60-65f4-4306-8753-d06a4fee97e0_isr_994640_30.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 4ea25aba757b0350b65b29f15603cfbc parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b865c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/70b24c54-5bbd-40b5-8a98-c9d36d348281_isr_994767_14.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/70b24c54-5bbd-40b5-8a98-c9d36d348281_isr_994767_14.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 2af3cf43db41e9cdbe44641d6339931d parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b87920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cf373f22-677e-497d-b1e7-7b23f445dc2b_isr_994639_12.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cf373f22-677e-497d-b1e7-7b23f445dc2b_isr_994639_12.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 4888a008b8fb1ea842710c5d22cd07a9 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b87600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/2eeddd72-9a51-495a-bc64-d9da823edd7f_isr_994638_15.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/2eeddd72-9a51-495a-bc64-d9da823edd7f_isr_994638_15.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 671827d6aae34ca22304ecec3fcdcb89 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b872e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e8ed626d-1a4f-438a-8806-ee3abc3cd3d3_isr_994640_46.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e8ed626d-1a4f-438a-8806-ee3abc3cd3d3_isr_994640_46.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 07295527f2333430b05a4a865b555015 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/d353f3d9-a316-4b88-908c-addd16da83ef_isr_994765_25.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/d353f3d9-a316-4b88-908c-addd16da83ef_isr_994765_25.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash c02bb23b61d9f2099e0efa897901bedf parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ee22638a-57cc-497f-af06-15c143386431_isr_994640_3.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ee22638a-57cc-497f-af06-15c143386431_isr_994640_3.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 59aa64463faa0b59bc394dae28ab1837 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b868e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3a266abe-fd4e-4e34-a3ac-de3798aef6c0_isr_994638_30.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3a266abe-fd4e-4e34-a3ac-de3798aef6c0_isr_994638_30.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash bc484de66ada741bbfd094cc6153c315 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b87ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d4565c2c-df15-4dd2-a327-7ea7e37f81aa_isr_994766_59.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d4565c2c-df15-4dd2-a327-7ea7e37f81aa_isr_994766_59.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash eb368e145d7a4e90c58949f50ddfe8ef parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cc3eca27-4f1d-4ffc-a1a8-789e5fc8d061_isr_994767_27.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cc3eca27-4f1d-4ffc-a1a8-789e5fc8d061_isr_994767_27.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 563ac828db84d6443dffe9d9e1e44750 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/dfe48501-854d-463e-ae7c-1a1cc3536e53_isr_994640_41.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/dfe48501-854d-463e-ae7c-1a1cc3536e53_isr_994640_41.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash ce98c3a51e5df43be1a1477265ce53fa parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/b6f43b29-61a0-40b3-8202-3d06d092dab9_isr_994767_30.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/b6f43b29-61a0-40b3-8202-3d06d092dab9_isr_994767_30.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 8efd6df190046133507c0698d1709628 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/898b6be7-ce81-4d31-ba4c-9f12bd1c8ed7_isr_994767_24.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/898b6be7-ce81-4d31-ba4c-9f12bd1c8ed7_isr_994767_24.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 3e53e04399b20e952bc854d8cede0158 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8ffc6d94-8a37-42b3-a710-c9a5956cdaca_isr_994766_36.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8ffc6d94-8a37-42b3-a710-c9a5956cdaca_isr_994766_36.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash eb849dae51ddefef3d638896ed748583 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ab1a351e-e0f8-40d6-8354-8b6ce61799b4_isr_994638_56.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ab1a351e-e0f8-40d6-8354-8b6ce61799b4_isr_994638_56.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 5607f504b8e32d8bec442aa8451f6e79 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/12ac353f-8a96-4e22-a42a-aaf376fb577b_isr_994767_57.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/12ac353f-8a96-4e22-a42a-aaf376fb577b_isr_994767_57.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 1ff1c267b9def039350eb1a46ca73b2a parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fe6f5f0d-ad8e-4f81-8fba-05e6c311cc4d_isr_994638_7.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fe6f5f0d-ad8e-4f81-8fba-05e6c311cc4d_isr_994638_7.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 88955add29bf24c87ab9c07dbe796b4a parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7033d20e-a0ba-4ea3-b19c-0389b2db2b68_isr_994767_40.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7033d20e-a0ba-4ea3-b19c-0389b2db2b68_isr_994767_40.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 2dba18e1e3d15b4208bbd32517c0a521 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3f2106e6-e635-4ed0-a015-ee1e25a6820f_isr_994640_20.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3f2106e6-e635-4ed0-a015-ee1e25a6820f_isr_994640_20.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash c9a580d5aba39bd2ec997178745c761f parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/14d95624-672e-4674-a471-037678a81501_isr_994765_2.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/14d95624-672e-4674-a471-037678a81501_isr_994765_2.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash d835ccbcf71be74d2b9152b5b5d2aa01 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/29a6d054-151c-4de3-a08f-af8f88543990_isr_994638_33.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/29a6d054-151c-4de3-a08f-af8f88543990_isr_994638_33.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 9c72c726455a749342fd948930190bf2 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/a24df6a5-be4a-47f7-812f-99dacd1be0d5_isr_994766_21.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/a24df6a5-be4a-47f7-812f-99dacd1be0d5_isr_994766_21.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash b7c75b1abd51f704b247b901d4e4999b parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/a2a18200-0342-450d-810a-d1a3d2a13028_isr_994639_21.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/a2a18200-0342-450d-810a-d1a3d2a13028_isr_994639_21.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 5f85d4c578cac048d1e51e6c6c1b351d parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/88cbbbd5-0c2d-4773-b30b-60a1fe1f6506_isr_994639_44.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/88cbbbd5-0c2d-4773-b30b-60a1fe1f6506_isr_994639_44.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 50c90bc88dbd1992b88b3634da7bc204 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/c122dce0-5bf9-4caa-9bed-448b0ae3a348_isr_994765_45.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/c122dce0-5bf9-4caa-9bed-448b0ae3a348_isr_994765_45.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash c1331000d1de51e49e58bba4939863a6 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5e320610-41d1-43c3-8120-746447e7a29e_isr_994767_10.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5e320610-41d1-43c3-8120-746447e7a29e_isr_994767_10.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 1d7fc75540d3994f38290b419684d5e4 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/eb66a2c3-4375-40eb-989f-994537f8013b_isr_994767_62.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/eb66a2c3-4375-40eb-989f-994537f8013b_isr_994767_62.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash b7483c41303092f147a70dacdb465574 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/9ea46b6d-37c3-4976-9862-c6c8fc94e4f9_isr_994766_31.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/9ea46b6d-37c3-4976-9862-c6c8fc94e4f9_isr_994766_31.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 6dd4c493ce552fdcfa144c4ff4408445 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d589d1a6-d39d-4f61-8c95-2b6566b0b59d_isr_994766_10.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d589d1a6-d39d-4f61-8c95-2b6566b0b59d_isr_994766_10.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 7678c1e9edadc7d1f24b6e002c02c4c4 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/62bb73cd-d163-4f91-a3d0-d08c9f1c943b_isr_994639_56.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/62bb73cd-d163-4f91-a3d0-d08c9f1c943b_isr_994639_56.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 4e49f6f07aa87dd8953c75221e14816a parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/b052eaab-3699-4ed7-946f-b819d31b7204_isr_994767_47.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/b052eaab-3699-4ed7-946f-b819d31b7204_isr_994767_47.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 7d71df77e91d3215ecc344c10f7532ac parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/60919b39-1912-4403-8355-67f22af220c2_isr_994639_52.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/60919b39-1912-4403-8355-67f22af220c2_isr_994639_52.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash a76f6719b18d9feee395942a48d167dc parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/03ba2c47-b350-483c-9d39-7dc91f208712_isr_994639_47.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/03ba2c47-b350-483c-9d39-7dc91f208712_isr_994639_47.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash e4dd4f6dde6614639ee084dc90a448d8 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d3e43342-7ca3-481a-a980-eb4ddf9547f7_isr_994767_29.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d3e43342-7ca3-481a-a980-eb4ddf9547f7_isr_994767_29.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash e04f36a5fc0d74b724921d51f5a8bd56 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/1f94dca6-518a-4046-8230-19a73571dec6_isr_994766_30.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/1f94dca6-518a-4046-8230-19a73571dec6_isr_994766_30.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 1625967d1316e3543c82d49f22a0f531 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/10329def-a9eb-493f-adfd-c4c6ecffcf05_isr_994766_49.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/10329def-a9eb-493f-adfd-c4c6ecffcf05_isr_994766_49.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash a6a753e6ba11825e28f4e108877372d0 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/421f41c9-74c2-4355-8840-74ddc3dcb652_isr_994766_26.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/421f41c9-74c2-4355-8840-74ddc3dcb652_isr_994766_26.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 63b07ed4272b2b878169ed9eaa243c75 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cc084427-0f25-4086-b7e8-e04c0bf37cb6_isr_994765_52.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cc084427-0f25-4086-b7e8-e04c0bf37cb6_isr_994765_52.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash ba6a16f9d92add0992ab10383cbaa0f2 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9228d7e2-6b2f-4de3-bc0d-52da472879e1_isr_994765_62.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9228d7e2-6b2f-4de3-bc0d-52da472879e1_isr_994765_62.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 5be0810d79b7ac9b84c986c0676dc1c7 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5dd1ce2b-16d5-46ab-945e-b5d4b7904bb5_isr_994640_62.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5dd1ce2b-16d5-46ab-945e-b5d4b7904bb5_isr_994640_62.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash fb0b274a521e99c39ce4da41770d56b5 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/0a1d8927-9458-4e81-bcb3-5927fc566d1b_isr_994639_22.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/0a1d8927-9458-4e81-bcb3-5927fc566d1b_isr_994639_22.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash eb167a4192cbdef4792af931c19cec3d parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/8a224f86-3d42-4ecd-a653-1e98370f82a2_isr_994639_11.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/8a224f86-3d42-4ecd-a653-1e98370f82a2_isr_994639_11.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 63a7c4c4effe4c4d4c51608faf866c8a parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ca208ad2-409f-47cb-962d-35e3073feaa8_isr_994638_59.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ca208ad2-409f-47cb-962d-35e3073feaa8_isr_994638_59.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 276b6e2b7a8c01a47f3cd93e715aa81d parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/c5722a1d-07b2-41b3-8bd8-8de2345a136c_isr_994638_17.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/c5722a1d-07b2-41b3-8bd8-8de2345a136c_isr_994638_17.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 03f49633298d1824648f5138c9c31028 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cd84988b-011f-4b8c-899c-360c90938ca6_isr_994767_25.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cd84988b-011f-4b8c-899c-360c90938ca6_isr_994767_25.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 54760103a59003ed21a04a758af33a13 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ce3211b2-aa50-436d-ac4d-2a105dc1fda5_isr_994766_19.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ce3211b2-aa50-436d-ac4d-2a105dc1fda5_isr_994766_19.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 14c529155eb15b1ec9790587a2961a66 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/039a5d61-4bcd-4d8e-b143-ace06fb92cd0_isr_994767_18.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/039a5d61-4bcd-4d8e-b143-ace06fb92cd0_isr_994767_18.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 2d6b2722dbff22c3b04aee5eec93b95f parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fa401e05-9ccf-40ad-86c7-1bc6e4b17b9d_isr_994765_5.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fa401e05-9ccf-40ad-86c7-1bc6e4b17b9d_isr_994765_5.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 70d548b3df830173aa4efca05d3993cf parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/bcf1240d-cf86-4b33-8f7f-06834f764f7d_isr_994639_28.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/bcf1240d-cf86-4b33-8f7f-06834f764f7d_isr_994639_28.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 68d1850b81ec63c80726083fcba13575 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/f9a9dba0-865e-4c32-b963-4b190f2b7f78_isr_994766_52.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/f9a9dba0-865e-4c32-b963-4b190f2b7f78_isr_994766_52.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 037dee1af6e674d65b2181f3a9f5e4c4 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/aa277727-0ec4-4ec7-a80d-6cab5dee82e7_isr_994766_47.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/aa277727-0ec4-4ec7-a80d-6cab5dee82e7_isr_994766_47.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash c980747289d13250952f2f6f26d33e22 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/27915628-10c8-4b09-bbef-7ae3c11060fa_isr_994765_60.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/27915628-10c8-4b09-bbef-7ae3c11060fa_isr_994765_60.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash f2f3884b8304a4441094f2432fbed4af parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/63217d78-3fbf-4f35-9993-ca9d7e75278f_isr_994639_14.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/63217d78-3fbf-4f35-9993-ca9d7e75278f_isr_994639_14.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 10a2ac4c1f0133f8c517120c92cadfe7 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/a9318e72-ac87-4f39-abae-304dfb671f5d_isr_994767_51.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/a9318e72-ac87-4f39-abae-304dfb671f5d_isr_994767_51.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 6dd3293bb7022ab1cab8cf8b75591f87 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87aa2a24-fadc-4f98-9aca-3fc287ac047f_isr_994765_27.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87aa2a24-fadc-4f98-9aca-3fc287ac047f_isr_994765_27.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 082d4115f9c04b4ada1b0fc173a91f13 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6f592b24-bcfe-4a8d-a754-4e936d8f14e1_isr_994638_48.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6f592b24-bcfe-4a8d-a754-4e936d8f14e1_isr_994638_48.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash a0e488d3d1723314f42a83f290a096e8 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/cf18ae22-edf4-41b7-be29-1a0e49ff6b98_isr_994766_7.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/cf18ae22-edf4-41b7-be29-1a0e49ff6b98_isr_994766_7.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 251cd960ce63604bfe5dcb25d3c7ed4e parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/27b97d78-c311-46d4-a9b2-19d3bb4abadd_isr_994639_31.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/27b97d78-c311-46d4-a9b2-19d3bb4abadd_isr_994639_31.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash 734f9ae000d771806a09f125eaa29e30 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0ce3815b-4af3-40cc-a7f5-db01efa43b25_isr_994765_57.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0ce3815b-4af3-40cc-a7f5-db01efa43b25_isr_994765_57.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash f666598ab2383cebdd767099d8817f97 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ec15d7d1-7e7d-4fe0-907e-33e1d3955a50_isr_994638_4.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ec15d7d1-7e7d-4fe0-907e-33e1d3955a50_isr_994638_4.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash c89205f02f219a9f203ea7021b7d3882 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ef4b5386-dc56-4461-aaf1-dde7b509de48_isr_994640_39.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ef4b5386-dc56-4461-aaf1-dde7b509de48_isr_994640_39.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 1a83760de46961b18ad9f48340e4026c parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/01471a38-6093-4ace-b1d3-9c127ccdab21_isr_994766_43.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/01471a38-6093-4ace-b1d3-9c127ccdab21_isr_994766_43.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash e39c08435370c7785b9db155cecd355c parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/6f135eb2-0613-41eb-acfb-6860ac3e6a20_isr_994767_16.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/6f135eb2-0613-41eb-acfb-6860ac3e6a20_isr_994767_16.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 98e508cd5af052e17aaee6cfa9e12798 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c8442a62-d3b4-402b-b700-e87b3e7b280f_isr_994767_4.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c8442a62-d3b4-402b-b700-e87b3e7b280f_isr_994767_4.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash d864f87a67029dd43d561a5b8f6dd48c parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/fec2937b-a826-441d-a583-8ef5389a6878_isr_994767_53.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/fec2937b-a826-441d-a583-8ef5389a6878_isr_994767_53.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 3d04ae40fe5f2c75670aa7318a0bc453 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d8f896c8-6d57-4c96-a0d3-22c9616794bc_isr_994639_53.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d8f896c8-6d57-4c96-a0d3-22c9616794bc_isr_994639_53.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash b28e7ec5c241d2cabb617e4ab50dba65 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7be9a731-982a-45ea-8b2b-ca5bdff61252_isr_994638_60.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7be9a731-982a-45ea-8b2b-ca5bdff61252_isr_994638_60.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 545b7ac2ac571e110d796a8eebc41d20 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8f3de220-1ce6-4bb4-82c5-846b94728bff_isr_994767_11.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8f3de220-1ce6-4bb4-82c5-846b94728bff_isr_994767_11.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 954bb4661153a2741660d50801051e12 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/22dde8c5-bfff-491c-925c-489650f7e8dd_isr_994766_60.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/22dde8c5-bfff-491c-925c-489650f7e8dd_isr_994766_60.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash aca97f5d1b55968e0a3f2df54ba34368 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7e41b692-1b2c-4892-8687-71d18a3d9517_isr_994638_36.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7e41b692-1b2c-4892-8687-71d18a3d9517_isr_994638_36.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 26f64692ba069c8e7423638d8c574af3 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/e5045079-dd41-481b-b5f3-143916e2f8b6_isr_994767_61.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/e5045079-dd41-481b-b5f3-143916e2f8b6_isr_994767_61.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 06df3f85ca39c8b318c7c5040e3e3ab3 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e42c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e4cc5dc6-fb2f-4b95-8b54-11be2860b4f1_isr_994640_58.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e4cc5dc6-fb2f-4b95-8b54-11be2860b4f1_isr_994640_58.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash e2042da792874e1401ec9771ef121daf parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e45e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/485a8ee6-30de-489f-bbbd-622a67e5e4d0_isr_994639_18.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/485a8ee6-30de-489f-bbbd-622a67e5e4d0_isr_994639_18.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 82efc9f334f5e7a7bae5704fe99fe86f parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3b826cce-41e7-48b3-9436-12cd1348043e_isr_994638_27.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3b826cce-41e7-48b3-9436-12cd1348043e_isr_994638_27.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash a70b795e575ae099f38b882bbc8a3a71 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d54c8b33-3abf-4889-a60d-3dbe91b43d6a_isr_994639_6.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d54c8b33-3abf-4889-a60d-3dbe91b43d6a_isr_994639_6.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash d535630f7194437965aa580c1bfd8ddf parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/0ec2e5b0-70f3-4824-956f-e711c18240d3_isr_994639_59.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/0ec2e5b0-70f3-4824-956f-e711c18240d3_isr_994639_59.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 15f3894ad24e1ab8ca607d4474b06be6 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e5260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/611c5128-6680-4e07-b582-8bcecb57a9a7_isr_994639_24.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/611c5128-6680-4e07-b582-8bcecb57a9a7_isr_994639_24.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 5cced51435d9ed47f558b1bfb44c363b parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e5580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c1992084-a1c0-4db7-ad1c-7584a392c410_isr_994767_34.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c1992084-a1c0-4db7-ad1c-7584a392c410_isr_994767_34.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 64e7215cf967dee4c4be90790f6cd623 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e58a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c8e0d60e-a6f1-47c6-bb0e-5b896d812397_isr_994640_4.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c8e0d60e-a6f1-47c6-bb0e-5b896d812397_isr_994640_4.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash d7806695c4970bfc5db91dcac91b1ced parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e5bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/5abc8802-22a0-4bb4-8f78-cf55c21c0c53_isr_994639_3.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/5abc8802-22a0-4bb4-8f78-cf55c21c0c53_isr_994639_3.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 27f844831e808d7737399a82338f0124 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e5ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e8430d30-5011-4eb6-ba79-da623e553428_isr_994640_43.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e8430d30-5011-4eb6-ba79-da623e553428_isr_994640_43.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 0bcf51c4b5dc078aaa353e049c6ab362 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/82b1bca8-07fb-4ad7-a137-9118e09d2b3e_isr_994766_20.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/82b1bca8-07fb-4ad7-a137-9118e09d2b3e_isr_994766_20.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 421773a13ccf9043259068a262417b7e parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/19cc2a90-1314-4516-9945-764708f2d164_isr_994766_18.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/19cc2a90-1314-4516-9945-764708f2d164_isr_994766_18.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash ad3e2fc6d90126675a37fb208f590bdb parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/390fc59b-7d5a-478d-9f8f-70e2deea4da8_isr_994767_37.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/390fc59b-7d5a-478d-9f8f-70e2deea4da8_isr_994767_37.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 9c98319d5d0dad355623c10a48d21a35 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/30904626-c451-4139-93da-adb9340bd04e_isr_994766_16.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/30904626-c451-4139-93da-adb9340bd04e_isr_994766_16.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash eb3f036e666bfe08c04b92c7d9a55f28 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fdde7ab5-f068-41b0-b0b1-0e211486bc70_isr_994638_29.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fdde7ab5-f068-41b0-b0b1-0e211486bc70_isr_994638_29.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash b8a14e6011fa73cc2ced9a03313a4d68 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e71a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5b719e66-c880-4d76-93bf-f64591371e6f_isr_994640_40.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5b719e66-c880-4d76-93bf-f64591371e6f_isr_994640_40.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 5f7250678283023a015b212d28eebbae parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e74c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/36a22edd-f226-4ff9-a3db-30be9e113260_isr_994640_52.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/36a22edd-f226-4ff9-a3db-30be9e113260_isr_994640_52.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 42edd5e9a793b594b20e9e5644d0f36e parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/623dc85f-6956-45e8-bb39-f262d0e202df_isr_994766_53.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/623dc85f-6956-45e8-bb39-f262d0e202df_isr_994766_53.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 5622bba8e3198259ad44fa9324262529 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/6b38b6dc-2371-4e1f-8cb9-ddd379235126_isr_994765_50.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/6b38b6dc-2371-4e1f-8cb9-ddd379235126_isr_994765_50.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 935b43ac28e2b149b0df4d346d44c81c parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e03e3aa5-6494-4b8c-84a8-6fa5194c2da5_isr_994640_53.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e03e3aa5-6494-4b8c-84a8-6fa5194c2da5_isr_994640_53.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 46d52481725a5da985d33d5f47b6de2e parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6600180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/a2dcad45-97d9-48b3-928e-b8172925d47f_isr_994639_16.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/a2dcad45-97d9-48b3-928e-b8172925d47f_isr_994639_16.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash b64fc47c45aed39b5da6e0ff4e0eeb2b parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66004a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7df265da-10ed-4f90-905a-8f9c19b60da5_isr_994767_55.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7df265da-10ed-4f90-905a-8f9c19b60da5_isr_994767_55.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 321f5649d12fc69597c7ec732d65bab9 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66007c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/e0d33581-2b45-4073-9e31-3062fb5a4676_isr_994766_11.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/e0d33581-2b45-4073-9e31-3062fb5a4676_isr_994766_11.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash a9c969e9bd69cfb45722d009c68833bd parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6600ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/01373479-0032-42cc-9542-0152d326ebf2_isr_994639_39.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/01373479-0032-42cc-9542-0152d326ebf2_isr_994639_39.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 67462de9bc3762f86c72e449254fa106 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6600e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9590beb9-2129-4a45-94f1-c236543ea00d_isr_994765_19.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9590beb9-2129-4a45-94f1-c236543ea00d_isr_994765_19.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 30301fb3a26261fd1a63d0f0848b48e2 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/89b8e7c6-7bb6-4842-ab54-cbbdbd293042_isr_994639_9.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/89b8e7c6-7bb6-4842-ab54-cbbdbd293042_isr_994639_9.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash dcc17175836bbe44966760036485fca6 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/2a135c37-2726-49ae-ada0-c93ae1eafd09_isr_994767_31.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/2a135c37-2726-49ae-ada0-c93ae1eafd09_isr_994767_31.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash be4e42dbb9cd1b5e9e3e91c690136da9 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ebdb1deb-eb35-42d7-8d57-03f16d42a393_isr_994767_60.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ebdb1deb-eb35-42d7-8d57-03f16d42a393_isr_994767_60.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 3c6573af43972f8da8221153eaa91124 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/958785ff-0498-43ee-8914-22dd15dc5af3_isr_994767_8.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/958785ff-0498-43ee-8914-22dd15dc5af3_isr_994767_8.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 4f1ebfd310bded22b1805e992ea088c9 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6db45845-4034-4b34-b77a-2b93a0a5c0ce_isr_994639_15.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6db45845-4034-4b34-b77a-2b93a0a5c0ce_isr_994639_15.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 1d07d3da78f7cb5761353a4d9dcdf016 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66020c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a00c1c83-c640-4eff-9bd7-31d6fb738a6d_isr_994765_34.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a00c1c83-c640-4eff-9bd7-31d6fb738a6d_isr_994765_34.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash b3fd93d92f93e96b476573f9406eeb4d parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66023e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/46f14f2b-234b-4439-9720-7d1d3180f0b6_isr_994640_44.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/46f14f2b-234b-4439-9720-7d1d3180f0b6_isr_994640_44.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash ee6a8d207743f5b2ff649533a7a6e0bb parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/add5ee37-2ba1-4909-baa3-035a13153a29_isr_994766_50.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/add5ee37-2ba1-4909-baa3-035a13153a29_isr_994766_50.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 22d3f89f462163a263b54885aead3335 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/2f95c209-cb14-40f5-9731-1bfbaa96eaed_isr_994639_62.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/2f95c209-cb14-40f5-9731-1bfbaa96eaed_isr_994639_62.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash 890d601ed29a00d7296c41835cc97350 parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cc0a214f-c6ed-44ff-8cec-4eda2dbada76_isr_994765_11.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cc0a214f-c6ed-44ff-8cec-4eda2dbada76_isr_994765_11.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash a9ed65c8fc0a31564266003de6689554 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6603060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/f48399ba-e7d1-4e81-9973-f11f935f5665_isr_994639_27.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/f48399ba-e7d1-4e81-9973-f11f935f5665_isr_994639_27.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 702bb5d069086b8c8b277565c0e84027 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6603380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/692f36c8-c0a6-434e-8006-c29bd1e99829_isr_994766_57.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/692f36c8-c0a6-434e-8006-c29bd1e99829_isr_994766_57.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 8f7f3bfe0f0d586365cf352887154983 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66036a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/1095c75f-9b25-407a-bbb7-3ecaeddd66e0_isr_994639_34.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/1095c75f-9b25-407a-bbb7-3ecaeddd66e0_isr_994639_34.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 9dbe282ad302ea9b7a2503485d9b3ea4 parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66039c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/955fc71c-86e6-4e18-8d88-6070a65b41dc_isr_994638_53.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/955fc71c-86e6-4e18-8d88-6070a65b41dc_isr_994638_53.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 81c3e26e3a3c36feb0fe1d94f749d861 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6603ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5d96ff8b-9a82-4591-8a4d-ef428c7d2e23_isr_994767_39.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5d96ff8b-9a82-4591-8a4d-ef428c7d2e23_isr_994767_39.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 627ba287687e1edd0854b430d18edaa3 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d82c0c64-6a0b-47a6-84c1-6cc647448d61_isr_994640_23.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d82c0c64-6a0b-47a6-84c1-6cc647448d61_isr_994640_23.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash b1688898cfae3474919250098fca4749 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/56092868-1d27-45bf-9546-3681595eacbc_isr_994640_14.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/56092868-1d27-45bf-9546-3681595eacbc_isr_994640_14.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash abb2c4d63a09d09e769eda19972ede7d parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/68902548-c7e9-432c-8098-e859658c8156_isr_994767_49.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/68902548-c7e9-432c-8098-e859658c8156_isr_994767_49.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 148386701b5beba803f0386d7589cbdc parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ce1b7ea9-bdff-41b8-b936-5d4b60df1cdd_isr_994639_32.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ce1b7ea9-bdff-41b8-b936-5d4b60df1cdd_isr_994639_32.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 379fbf225477aa06dd0d960fb69f3697 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f29a70e6-b19b-4f06-9441-b5637175a26e_isr_994640_13.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f29a70e6-b19b-4f06-9441-b5637175a26e_isr_994640_13.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash ae74c21818097139e2d86bc75d4a0984 parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6c7d34e5-170e-41f0-8e2c-1030fe415b4b_isr_994639_48.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6c7d34e5-170e-41f0-8e2c-1030fe415b4b_isr_994639_48.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash ff11aab5189e7c6fa46d5d5227ca7704 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:13:02 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/4d65e774-3d13-495c-bbab-b34a52de1ffb_isr_994765_16.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/4d65e774-3d13-495c-bbab-b34a52de1ffb_isr_994765_16.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash c31f129ed6bc4e079439241e1e50f2c0 parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/38b12774-aef3-4efa-9b05-b02694beb311_isr_994639_60.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/38b12774-aef3-4efa-9b05-b02694beb311_isr_994639_60.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash 49659ec53717ff42b1f45943ee926e6f parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/705bd872-743c-4447-a793-f5f927beb0a9_isr_994640_1.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/705bd872-743c-4447-a793-f5f927beb0a9_isr_994640_1.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 253e78290b8dcf50d8494ad539b0d724 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/222e2ccf-a16f-4f6d-a78d-46095046f79a_isr_994765_41.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/222e2ccf-a16f-4f6d-a78d-46095046f79a_isr_994765_41.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 3701874955241b129355308b071b2510 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/bf529c42-cdef-4021-971b-3a7fc35629db_isr_994638_8.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/bf529c42-cdef-4021-971b-3a7fc35629db_isr_994638_8.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash cf554339a0701b025fa6d92bbb6e3630 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/50e6c21a-e104-43ba-a467-c29c98f76533_isr_994639_5.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/50e6c21a-e104-43ba-a467-c29c98f76533_isr_994639_5.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash df58a6ac312a0e77c988f97b888b719e parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 130 parsl.dataflow.strategy DEBUG: Executor multi has 130 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 32 more blocks parsl.executors.status_handling INFO: Scaling out by 32 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-15 04:13:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-15 04:13:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 04:13:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/0638be76-00f8-4130-bcbe-c7c58da51997_isr_994767_36.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/0638be76-00f8-4130-bcbe-c7c58da51997_isr_994767_36.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash e41d16c666ddbe7c0468e4c495f32dba parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9d9d2f3c-0d15-44db-8e7a-6f93015d141c_isr_994767_32.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9d9d2f3c-0d15-44db-8e7a-6f93015d141c_isr_994767_32.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 168791fa61b44d0afd021fd9deacf140 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/85cc45bd-f32a-4c04-93de-6a157381d2da_isr_994765_20.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/85cc45bd-f32a-4c04-93de-6a157381d2da_isr_994765_20.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 44e4a02c4ba1dec3f649d3e15d3f5914 parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66522a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f773fce6-1aa2-4676-9519-1ecc1f386f2d_isr_994640_35.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f773fce6-1aa2-4676-9519-1ecc1f386f2d_isr_994640_35.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 0bb0c6b0c6598da8e0094d0cbe9919b4 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6651f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5cb35976-1065-4111-9a7c-4ff1de95ca7e_isr_994640_34.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5cb35976-1065-4111-9a7c-4ff1de95ca7e_isr_994640_34.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 264bab75030ae69c73e02cd7916d6afc parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6651c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/40dc4367-1d2a-4b02-b3ed-76f18b4052ff_isr_994639_36.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/40dc4367-1d2a-4b02-b3ed-76f18b4052ff_isr_994639_36.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 6160c773c02b476e5253c35007decb62 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d7b53daf-8d68-46fe-b5d6-e8e0dfcb72fe_isr_994639_51.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d7b53daf-8d68-46fe-b5d6-e8e0dfcb72fe_isr_994639_51.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 67e82d121a297b07170d50e96602d881 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ce822876-e0eb-4a86-b564-393175aea6c2_isr_994767_9.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ce822876-e0eb-4a86-b564-393175aea6c2_isr_994767_9.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash aa1eb39e6f9222aec6474bad3afbcf16 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c9571dda-dcae-4ea9-b966-9de27dfeddb3_isr_994640_37.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c9571dda-dcae-4ea9-b966-9de27dfeddb3_isr_994640_37.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash 7f97a94308b454d7a2294ebaff54a085 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b4e44f8f-4e91-40ef-8652-0c4ba03e1399_isr_994766_13.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b4e44f8f-4e91-40ef-8652-0c4ba03e1399_isr_994766_13.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 7ee8ec9229d88bf3b0e2773a2d991a25 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6651d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2a407662-e142-41bc-9e8f-bea15f982d55_isr_994766_35.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2a407662-e142-41bc-9e8f-bea15f982d55_isr_994766_35.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash f0b2823a6a3ba3c0514d5bc3d64ef8c3 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/95ab9ba8-9bf3-4037-a1d2-c45b21c9c162_isr_994765_31.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/95ab9ba8-9bf3-4037-a1d2-c45b21c9c162_isr_994765_31.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 7a7e1c1855a51918802346b097d9b3fb parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8a5b24aa-03a3-42d1-91c0-fa5c8d8e9499_isr_994767_43.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8a5b24aa-03a3-42d1-91c0-fa5c8d8e9499_isr_994767_43.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash b6dcbedc2885f48d36db61d0d37c124a parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66507c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/b3f99e42-d278-4f1c-a5e0-16c789af7a3d_isr_994639_46.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/b3f99e42-d278-4f1c-a5e0-16c789af7a3d_isr_994639_46.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash c5514884108304050f629d23386d0873 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/17c9b672-695e-4e94-9666-a57ce13ac6a7_isr_994767_54.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/17c9b672-695e-4e94-9666-a57ce13ac6a7_isr_994767_54.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 5cbdbc891db51309caeb566e3df2fd43 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8ae0ac5c-aeb2-433c-96b3-b281ec4a794e_isr_994766_3.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8ae0ac5c-aeb2-433c-96b3-b281ec4a794e_isr_994766_3.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 4c9ef99a66d32d23f216ab085dbf8320 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6651940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6bc82448-70e0-4d55-9575-53d88150608e_isr_994639_26.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6bc82448-70e0-4d55-9575-53d88150608e_isr_994639_26.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash 864621b5dda14265a3e69dc425e9337d parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6651620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/7f73d2fe-61bf-4246-b52a-649a93e46109_isr_994766_2.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/7f73d2fe-61bf-4246-b52a-649a93e46109_isr_994766_2.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash 0fbaaaaf024583b623e75f30bceb37c3 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6651300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ae3e1920-dfb0-42fb-b25f-b4793f95435b_isr_994639_7.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ae3e1920-dfb0-42fb-b25f-b4793f95435b_isr_994639_7.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash 8515e6b31fe0b8d9269e5f386f6bfb83 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6652520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2374bf25-db36-416c-8d8c-fe8b1fcfcffd_isr_994766_61.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2374bf25-db36-416c-8d8c-fe8b1fcfcffd_isr_994766_61.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 7013691b4fd40ac215c0987943cb05ba parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6652840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/19b9d3dc-ead4-4049-8fba-68a6fc761138_isr_994640_57.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/19b9d3dc-ead4-4049-8fba-68a6fc761138_isr_994640_57.stderr 2024-03-15 04:13:02 proc_lsst.multi:170 [INFO] [multi] job_id 17057287 proc_lsst.multi INFO: [multi] job_id 17057287 2024-03-15 04:13:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057287 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057287 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17057287 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-15 04:13:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-15 04:13:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash 9ed9a8ae02dd208fbaeb4c8a709a4b0f parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66528e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/3f65deb9-0d94-4d9a-bdbe-b52c749b5eb3_isr_994766_14.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/3f65deb9-0d94-4d9a-bdbe-b52c749b5eb3_isr_994766_14.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash ad719b6d6d777ba489ef8e8426748a8c parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6652ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/f66a2d99-61ec-4d55-8cdd-c20149e9a8b1_isr_994765_54.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/f66a2d99-61ec-4d55-8cdd-c20149e9a8b1_isr_994765_54.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash da90b260bb5af5ea1d88758d0fdfb28d parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64863e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6f66111e-1fbe-4297-8fa1-f7e296b5c6ca_isr_994638_54.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6f66111e-1fbe-4297-8fa1-f7e296b5c6ca_isr_994638_54.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash cb8bb71d1a114a4568725e89ff0e1f40 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64860c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/15b2c357-e0b3-462c-880a-4f374ff1da39_isr_994765_7.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/15b2c357-e0b3-462c-880a-4f374ff1da39_isr_994765_7.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 81d9e40ee7d9dba3dffc9cf96a7957a7 parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/17d55cc3-ab57-4e53-8501-f69573fa1903_isr_994766_51.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/17d55cc3-ab57-4e53-8501-f69573fa1903_isr_994766_51.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash a5b076cf8400f62123ee90a028ffdedd parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cb41f1b5-e7a2-47b3-a714-2b4f060f55e0_isr_994765_9.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cb41f1b5-e7a2-47b3-a714-2b4f060f55e0_isr_994765_9.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 82948838639926b46abd9ef46644dc3a parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/f2349455-c72c-4919-9069-df159487babe_isr_994767_21.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/f2349455-c72c-4919-9069-df159487babe_isr_994767_21.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash c1ec88679ce84544ccb071b80b18a99c parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fcf79596-189b-48f1-984b-f76e0a92ec04_isr_994765_35.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fcf79596-189b-48f1-984b-f76e0a92ec04_isr_994765_35.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash fc3bfcb7cbd22f9bf70351ce897d6519 parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e2230cf2-038e-46eb-80ea-51b0758eec42_isr_994640_54.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e2230cf2-038e-46eb-80ea-51b0758eec42_isr_994640_54.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash bee7c25747d10d77592175910e059829 parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a7e9afde-d6fa-438c-a89d-1d77350c774b_isr_994765_42.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a7e9afde-d6fa-438c-a89d-1d77350c774b_isr_994765_42.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash 95be6f089a66c69977df0aa034d01b10 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6709cc75-74f0-4e1e-a6f1-3e08ce9c12d3_isr_994639_40.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6709cc75-74f0-4e1e-a6f1-3e08ce9c12d3_isr_994639_40.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 42fa655c376eff2bea82267954a81416 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64844a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/4ea3c15a-ff84-418f-84fb-31ca52c5df55_isr_994767_41.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/4ea3c15a-ff84-418f-84fb-31ca52c5df55_isr_994767_41.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash 850fdd4eb604b344dcdac93322208af9 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/8907bb10-f2a9-472f-8e0c-2b86fef05248_isr_994640_61.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/8907bb10-f2a9-472f-8e0c-2b86fef05248_isr_994640_61.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash cfd6b22f2ea2d3a8bc6e10f1064820b0 parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/7420c4d5-a3a4-4732-a433-01770c58e23d_isr_994766_38.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/7420c4d5-a3a4-4732-a433-01770c58e23d_isr_994766_38.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 4b63fa889abe85826fd32e6277c929ff parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/efc5b6eb-5481-4d36-8e01-45d739531a0f_isr_994767_19.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/efc5b6eb-5481-4d36-8e01-45d739531a0f_isr_994767_19.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash 626dcdeb60787c746b2789c33313a4ac parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fd900e0a-7fb8-49ab-957c-8869a0aca665_isr_994638_39.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fd900e0a-7fb8-49ab-957c-8869a0aca665_isr_994638_39.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 9f0e5e478c49952dd5cb33cdefa465e2 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/bb50f2bb-36a2-4c79-9874-e0808e009809_isr_994767_2.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/bb50f2bb-36a2-4c79-9874-e0808e009809_isr_994767_2.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash 92d03fb421460be5897da3135430f262 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/1ca2c971-4e1d-46ff-9a7f-f0c58a0edc02_isr_994638_31.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/1ca2c971-4e1d-46ff-9a7f-f0c58a0edc02_isr_994638_31.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash d7daa2fba2e0d9bf7dafdf62dd989087 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/21be9294-61fc-4d9c-b380-9f2df20ec71a_isr_994638_34.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/21be9294-61fc-4d9c-b380-9f2df20ec71a_isr_994638_34.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash c1e5836f5292325d3a1c242b05dcc0a7 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bf10c3a4-80f8-423f-a1be-435fdf6cc9e0_isr_994766_55.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bf10c3a4-80f8-423f-a1be-435fdf6cc9e0_isr_994766_55.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash 1d602b8311d15d44b99842b5bc914c50 parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ce89154c-ccf6-4b3d-8611-7069b8b2b1f3_isr_994640_15.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ce89154c-ccf6-4b3d-8611-7069b8b2b1f3_isr_994640_15.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 2676726aca938a638a2cfe8c00e252c0 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/6af1e63c-3f68-4792-9c97-d31ea6739f64_isr_994766_58.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/6af1e63c-3f68-4792-9c97-d31ea6739f64_isr_994766_58.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 516 has memoization hash acd0accb9ebee11f54541f650a1a6f76 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6487060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:13:02 proc_lsst.multi:170 [INFO] [multi] job_id 17057288 proc_lsst.multi INFO: [multi] job_id 17057288 2024-03-15 04:13:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 04:13:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057288 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057288 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17057288 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-15 04:13:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-15 04:13:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 04:13:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/611ea10c-28b6-4ca5-9bf0-3595b8a716d4_isr_994638_6.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/611ea10c-28b6-4ca5-9bf0-3595b8a716d4_isr_994638_6.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash 861f762d7efbeb02b0f88f8f215fa359 parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6487100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/04d1f5e0-9162-4a19-a96a-e568ccdc8b4c_isr_994640_32.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/04d1f5e0-9162-4a19-a96a-e568ccdc8b4c_isr_994640_32.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash 7faadff62b5057329d37b0b821e8943f parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64874c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e5e457fa-6bba-435c-8346-fcb85e3be101_isr_994640_17.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e5e457fa-6bba-435c-8346-fcb85e3be101_isr_994640_17.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash af92ca857ac2685fcb01a1e064139d71 parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6487880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9860b7ef-0a2f-4fd1-a60e-a679cca60d77_isr_994767_46.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9860b7ef-0a2f-4fd1-a60e-a679cca60d77_isr_994767_46.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash 98aaaa1b99e46c4fb797f10a5f8127ba parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6487ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/c2cda851-efcf-4612-addc-684f8200eb67_isr_994765_37.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/c2cda851-efcf-4612-addc-684f8200eb67_isr_994765_37.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash bd3290ebcb2fd2fe06501e34fb1f485d parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64befc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/88f7f43f-a12e-4f53-9182-7aac0b44b170_isr_994638_42.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/88f7f43f-a12e-4f53-9182-7aac0b44b170_isr_994638_42.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash 60bb53bc2ec01295f39ecdf4996376cd parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64beca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/be6d9d0b-5cd3-4a22-98e4-c884fbc2baef_isr_994767_23.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/be6d9d0b-5cd3-4a22-98e4-c884fbc2baef_isr_994767_23.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash 5e0b414ec58aa8a4001d9e9e7fc83e12 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64be980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8fad4a9a-0f4b-45d0-9b78-790414eaad90_isr_994638_22.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8fad4a9a-0f4b-45d0-9b78-790414eaad90_isr_994638_22.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 537540529a9511dcbdf0bfb6e5f092d3 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/555b860b-59e1-4e42-9d2c-9a86d1d6922a_isr_994766_37.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/555b860b-59e1-4e42-9d2c-9a86d1d6922a_isr_994766_37.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 23340b95e56c4e52e29994b8f75f34aa parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/23782b8c-97fe-417b-b42b-bace24f72074_isr_994640_22.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/23782b8c-97fe-417b-b42b-bace24f72074_isr_994640_22.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash 9499d0c5e2c0df2990f12d8c6c706b73 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b60b0587-3042-4463-9b85-87e77d07b115_isr_994765_39.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b60b0587-3042-4463-9b85-87e77d07b115_isr_994765_39.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 1fd41111dab3202ec857897c56823a30 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/af870c7d-b995-47d3-b123-7f761c23bda0_isr_994765_4.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/af870c7d-b995-47d3-b123-7f761c23bda0_isr_994765_4.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash a371beda1e3b89f9a0e2cbc3bd34e873 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/897bf95c-12d7-4b74-ae15-8b8f981fe8a3_isr_994638_47.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/897bf95c-12d7-4b74-ae15-8b8f981fe8a3_isr_994638_47.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash b6b7aa5fa00b4858060938e6b56c2ee5 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/4d340158-6fb8-4be3-a06b-d63f39e35d3e_isr_994639_8.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/4d340158-6fb8-4be3-a06b-d63f39e35d3e_isr_994639_8.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash 9b65827959fa99f126dd34a86926456e parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0ee59462-217f-4f28-a0f6-2c86b85b5b2a_isr_994765_14.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0ee59462-217f-4f28-a0f6-2c86b85b5b2a_isr_994765_14.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash 478be3ada8194b2b63b30ca5ca2626c3 parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/30329358-91c6-4f50-a61f-2eca1c0881e2_isr_994766_15.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/30329358-91c6-4f50-a61f-2eca1c0881e2_isr_994766_15.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 32c2b4630dffefdb87738b59453eb952 parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/32544663-ce9d-4489-95d5-7139c45e1821_isr_994640_45.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/32544663-ce9d-4489-95d5-7139c45e1821_isr_994640_45.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 0b93ba7038945bb6c1e5ef6965aa9b0f parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/31ac6abc-9a5f-443a-aec3-0be283e4b2a3_isr_994638_40.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/31ac6abc-9a5f-443a-aec3-0be283e4b2a3_isr_994638_40.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash 94fac503517bc2a5873199ffd717ac70 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64be660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/e1c87191-863a-4936-9845-3807a6d75aa9_isr_994767_35.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/e1c87191-863a-4936-9845-3807a6d75aa9_isr_994767_35.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash d5e0feb5a8f00273a4081783492bc927 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64be340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/19dd2d23-7d9c-461e-ba24-1a8dc92a2b9b_isr_994766_48.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/19dd2d23-7d9c-461e-ba24-1a8dc92a2b9b_isr_994766_48.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash 9ed728379c0c74435e8e8d1979e77cda parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64be020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3a8f288f-64bb-467f-9a77-aa420357a090_isr_994640_18.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3a8f288f-64bb-467f-9a77-aa420357a090_isr_994640_18.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 0bf4350142b1471879592f8e2ab512ce parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b873dd31-7f91-42e4-bc4a-2c70346cbd48_isr_994766_54.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b873dd31-7f91-42e4-bc4a-2c70346cbd48_isr_994766_54.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 90f282d56c67a98b4277164ef7dbed37 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/91231bb5-87a2-4654-aec1-7c870a2fe1d2_isr_994640_2.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/91231bb5-87a2-4654-aec1-7c870a2fe1d2_isr_994640_2.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash ab0cdb658346035566cef26509e03605 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/b31ded7d-c8d8-4c3b-918c-e327d1591a96_isr_994640_51.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/b31ded7d-c8d8-4c3b-918c-e327d1591a96_isr_994640_51.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash de7a883431f019a93f7a819782b7bc8a parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5e1459bc-e985-49f9-9df7-0f2d3f79bcbc_isr_994640_21.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5e1459bc-e985-49f9-9df7-0f2d3f79bcbc_isr_994640_21.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 1857b15b375b72f1723fcbc00705e221 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d4319505-1367-463d-8341-a060c9f4f27f_isr_994767_1.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d4319505-1367-463d-8341-a060c9f4f27f_isr_994767_1.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash 9b134e63576f66892500f18cf161df49 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/466ecde4-095b-4cd1-95db-9744a7bb5606_isr_994639_2.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/466ecde4-095b-4cd1-95db-9744a7bb5606_isr_994639_2.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash 267fe833b7587bc86428fc9b3fc33714 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:13:03 proc_lsst.multi:170 [INFO] [multi] job_id 17057289 proc_lsst.multi INFO: [multi] job_id 17057289 2024-03-15 04:13:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 04:13:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057289 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057289 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17057289 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-15 04:13:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-15 04:13:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 04:13:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/460b9fbd-13b9-44b7-accf-9cefd44fc6d9_isr_994639_33.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/460b9fbd-13b9-44b7-accf-9cefd44fc6d9_isr_994639_33.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 94754bbfb84ffa50d2af3b6b3571a87d parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65045e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/a973a03d-64d3-419f-9ba2-c4b2b0b1578e_isr_994766_39.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/a973a03d-64d3-419f-9ba2-c4b2b0b1578e_isr_994766_39.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash bdfed551a071688a485f338c8701192b parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65049a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d713747a-76c5-46c9-aa1d-f1bb546b5e40_isr_994640_31.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d713747a-76c5-46c9-aa1d-f1bb546b5e40_isr_994640_31.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 84dab25477fda71720f6062b9c7b9c6d parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/0fe5f965-7266-4ce3-a9fd-be5965827da6_isr_994638_16.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/0fe5f965-7266-4ce3-a9fd-be5965827da6_isr_994638_16.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash 77f289653ede9b4652bad18191e72d08 parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6507c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5a37a655-6382-4314-b6d1-ff4ac4682415_isr_994640_59.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5a37a655-6382-4314-b6d1-ff4ac4682415_isr_994640_59.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash de948deaeda3e7f8125e59336b477008 parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6507920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/27e3bd17-1ef1-48cd-9193-1f7086eef65e_isr_994640_8.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/27e3bd17-1ef1-48cd-9193-1f7086eef65e_isr_994640_8.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 114657199e763934f3f32d488c097ef5 parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6507600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f3b4a8d3-66af-4607-864f-f3f76f52845c_isr_994638_24.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f3b4a8d3-66af-4607-864f-f3f76f52845c_isr_994638_24.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash f39f27b258516c7f431e6d0df513f262 parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ac66bcc3-130c-46b5-98a0-ee1f55b0a6ff_isr_994639_50.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ac66bcc3-130c-46b5-98a0-ee1f55b0a6ff_isr_994639_50.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 6fb955b834b9391783b98196d198c3c4 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/eb7fcd46-abb2-4718-8923-58f7bf8b4bd6_isr_994765_38.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/eb7fcd46-abb2-4718-8923-58f7bf8b4bd6_isr_994765_38.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash e3f40a2f93c2c1a321b2117f0ff719a8 parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cd4e75c5-4733-4945-b8dd-9124ba186274_isr_994639_54.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cd4e75c5-4733-4945-b8dd-9124ba186274_isr_994639_54.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash c8893b4779b34d2e004d83317d1afebc parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4cfe4753-1e27-49ab-8c24-413af7ebff68_isr_994766_32.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4cfe4753-1e27-49ab-8c24-413af7ebff68_isr_994766_32.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 11810f6670f7ec458087469545fe8af5 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3a470598-b014-4d52-877c-81c521a11a33_isr_994640_9.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3a470598-b014-4d52-877c-81c521a11a33_isr_994640_9.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash f795fe2c4d8f91fcb2dfe1d4abcfa8c5 parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6b946787-dae8-4eee-9042-ae63f0cdd54c_isr_994638_20.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6b946787-dae8-4eee-9042-ae63f0cdd54c_isr_994638_20.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash f82722fecb814ca72c0396486f904ead parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6506200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/3ca75a10-8391-43c7-bfb9-a9faa14b3e8e_isr_994765_18.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/3ca75a10-8391-43c7-bfb9-a9faa14b3e8e_isr_994765_18.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 5da7725739e52de5730a6c2707baeccd parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65065c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b0d45f28-e641-48fb-83a2-29cd1729c4e3_isr_994765_51.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b0d45f28-e641-48fb-83a2-29cd1729c4e3_isr_994765_51.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash 483d4f32517047e4887427b40d6c4146 parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65068e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/da849c32-9bbd-427b-a3bf-28a5eb7158da_isr_994640_6.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/da849c32-9bbd-427b-a3bf-28a5eb7158da_isr_994640_6.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash 17755b3949ed5b1ba194ba819142e3aa parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6506980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/85b27f1c-550e-4b7f-b954-30ea1817b8cf_isr_994765_22.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/85b27f1c-550e-4b7f-b954-30ea1817b8cf_isr_994765_22.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 3741504585ce19ba755ecef67f18fcee parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65072e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d8405de3-8a8a-4dd8-a589-aa3a6616c5f9_isr_994766_25.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d8405de3-8a8a-4dd8-a589-aa3a6616c5f9_isr_994766_25.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash b14f776a238b544c93db30c59ce15b74 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6506fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/bc83f787-850d-4ada-8318-458bbf9c3601_isr_994638_14.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/bc83f787-850d-4ada-8318-458bbf9c3601_isr_994638_14.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash c768b3ca574e9d59545e494ae699cbf8 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/fd00cde7-abca-42e7-9e98-6d8037b95590_isr_994639_41.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/fd00cde7-abca-42e7-9e98-6d8037b95590_isr_994639_41.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash acdb21148eba380cf32a858219f5ace0 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/86b68bef-2b74-4d19-8324-810056343420_isr_994767_17.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/86b68bef-2b74-4d19-8324-810056343420_isr_994767_17.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash c165332bd818ef09689e45b251da1752 parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7aa9d160-70bb-42e8-a6af-e8e58e2de5ae_isr_994638_10.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7aa9d160-70bb-42e8-a6af-e8e58e2de5ae_isr_994638_10.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 7dafae5ee7a7587be0031b9ab527bd7b parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/7933944d-eaaa-4932-93c1-f2b5dddc1b60_isr_994639_57.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/7933944d-eaaa-4932-93c1-f2b5dddc1b60_isr_994639_57.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash c247927934045e1e2ff03f81062f8131 parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:13:03 proc_lsst.multi:170 [INFO] [multi] job_id 17057290 proc_lsst.multi INFO: [multi] job_id 17057290 2024-03-15 04:13:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 04:13:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057290 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057290 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17057290 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-15 04:13:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-15 04:13:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 04:13:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/530c85d5-64c4-48c9-8e74-8424bf0b06b1_isr_994640_25.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/530c85d5-64c4-48c9-8e74-8424bf0b06b1_isr_994640_25.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 748080e00b4e190ac2d9a3fa11c08819 parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/7f99a35b-fc65-43c1-87c2-ee5f75942eed_isr_994765_10.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/7f99a35b-fc65-43c1-87c2-ee5f75942eed_isr_994765_10.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash cb6c00fb53d58393dbc085e33e9d398d parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/611dd07c-64ff-4593-9eb2-26150228486a_isr_994765_28.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/611dd07c-64ff-4593-9eb2-26150228486a_isr_994765_28.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash 86fec7fe42df64f23ab3e18caf986039 parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/efaa96a5-8dec-47a6-bdad-34ef5a96e7be_isr_994766_56.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/efaa96a5-8dec-47a6-bdad-34ef5a96e7be_isr_994766_56.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 60285cac25fad7af6f25c50a4c8b1335 parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c8363a36-05fe-482d-813d-0a37d2391b95_isr_994767_20.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c8363a36-05fe-482d-813d-0a37d2391b95_isr_994767_20.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash bf688d871492d2407cf6e20f6a10bfb0 parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/4fd0c220-087b-4a04-8a6b-5b651257c4fc_isr_994638_21.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/4fd0c220-087b-4a04-8a6b-5b651257c4fc_isr_994638_21.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash ad3f93ca59e32d6bad775e7195f7d1ae parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/e08f7824-71f8-4d1f-b16e-854c4a1d9bfe_isr_994765_40.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/e08f7824-71f8-4d1f-b16e-854c4a1d9bfe_isr_994765_40.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash 4c2bc0ae5bd7289d4ea2f197d1572804 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/0aab6239-7f5f-4f11-ba42-e9e52cd824a5_isr_994640_50.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/0aab6239-7f5f-4f11-ba42-e9e52cd824a5_isr_994640_50.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash eae29897d3aa437b45608d154699d6f1 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/217977e2-6180-4e0c-ab5a-aeaa697f8cd0_isr_994765_6.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/217977e2-6180-4e0c-ab5a-aeaa697f8cd0_isr_994765_6.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash b3567bb08747b2ac3d11ed47427de77d parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/bae3303a-66a2-45cc-b78b-b677b3547eb7_isr_994640_16.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/bae3303a-66a2-45cc-b78b-b677b3547eb7_isr_994640_16.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash 4546a1a9b3b94455bf85171de000ecd3 parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/0dd62aea-726c-414c-ba53-27383b90aae9_isr_994767_58.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/0dd62aea-726c-414c-ba53-27383b90aae9_isr_994767_58.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash f7fa29459c4ed1c3eb7c9fd734b8e7f6 parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/2c30a0dc-2b00-4615-a4a7-6eead8027c9c_isr_994640_49.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/2c30a0dc-2b00-4615-a4a7-6eead8027c9c_isr_994640_49.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 8df9875dd7e0e5fd9200c6a799be5c5c parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2c20c85f-e5d1-45f7-9b93-5db1aba4158e_isr_994766_5.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2c20c85f-e5d1-45f7-9b93-5db1aba4158e_isr_994766_5.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash de7445a2b08d87ac7046ec0a940b8661 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/62d2426d-a441-4e05-b2ef-43e58ba15514_isr_994638_5.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/62d2426d-a441-4e05-b2ef-43e58ba15514_isr_994638_5.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 3b548b7f685d3d4d464d796444bf1f93 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4347b490-3351-4905-a902-3b7393d1cd2c_isr_994766_12.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4347b490-3351-4905-a902-3b7393d1cd2c_isr_994766_12.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 8ea88d802b68fb4fa3aac7d055ebd66b parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:13:03 proc_lsst.multi:170 [INFO] [multi] job_id 17057291 proc_lsst.multi INFO: [multi] job_id 17057291 2024-03-15 04:13:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057291 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057291 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17057291 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-15 04:13:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-15 04:13:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/cbf5b206-0469-4ae2-97fa-5460d1fc3e26_isr_994640_29.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/cbf5b206-0469-4ae2-97fa-5460d1fc3e26_isr_994640_29.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash a660376fff4b56e507acaab9ccb8c311 parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/be831f52-feb1-49bd-8d1c-0442cf909755_isr_994766_17.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/be831f52-feb1-49bd-8d1c-0442cf909755_isr_994766_17.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 787099d21d892671990946f0f6fdbf46 parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/10e3ef23-f34f-4199-8832-664c90ea80a8_isr_994639_20.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/10e3ef23-f34f-4199-8832-664c90ea80a8_isr_994639_20.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash b0868d867527eb9ea61dff50b54877af parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7c695369-0353-4861-9f68-f5de52f83a50_isr_994767_28.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7c695369-0353-4861-9f68-f5de52f83a50_isr_994767_28.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash 0536db1f10d034106269ed0cb6f65c01 parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/fb3cb49e-9392-4eeb-a4f7-4bfa46996df9_isr_994640_24.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/fb3cb49e-9392-4eeb-a4f7-4bfa46996df9_isr_994640_24.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 0732f1fcca36e9f18d3ebe0667c3242e parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/da9cc97d-91d0-4f78-83f5-0612011a319d_isr_994765_59.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/da9cc97d-91d0-4f78-83f5-0612011a319d_isr_994765_59.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash 60e4cf59105d659da7f8adf3ab29ec03 parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/42bcf247-aaaa-4823-8d8b-9cdf870aac6f_isr_994640_27.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/42bcf247-aaaa-4823-8d8b-9cdf870aac6f_isr_994640_27.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 32f5f577428f8e537c1ff4cd43bb3f19 parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a675da93-565d-48a5-8ffe-e257fc21204e_isr_994765_3.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a675da93-565d-48a5-8ffe-e257fc21204e_isr_994765_3.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash fc855050a87f524dea3e48ee13e8bd45 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/d68ca3a6-605f-4c4f-af7b-52e1d88cbaf7_isr_994765_26.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/d68ca3a6-605f-4c4f-af7b-52e1d88cbaf7_isr_994765_26.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash 3a7838e7313d1f6bf25bfc71031f1685 parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87084e17-5e27-4858-b035-b7b56ddbd2c8_isr_994765_44.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87084e17-5e27-4858-b035-b7b56ddbd2c8_isr_994765_44.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 44f0a978bd772e29f2b2ddb3519480fc parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/8812509c-3e56-4352-b3a2-99e1d13f8301_isr_994639_45.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/8812509c-3e56-4352-b3a2-99e1d13f8301_isr_994639_45.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash f0bf12c855e349cad1a553fb9c1500ac parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/2266382e-5d29-4f33-91d0-a96fdbc73c13_isr_994639_1.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/2266382e-5d29-4f33-91d0-a96fdbc73c13_isr_994639_1.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App characterizeImage, waiting on task 744 parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 745 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App calibrate, waiting on task 745 parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 746 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash 5f86cc135febc98117f1c2523b3a9d6c parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6392e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cf705f0d-8331-4fba-91f9-29b9c66737b0_isr_994767_42.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cf705f0d-8331-4fba-91f9-29b9c66737b0_isr_994767_42.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App characterizeImage, waiting on task 747 parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 748 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App calibrate, waiting on task 748 parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 749 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash c36f1bf8cbb334762ccc06972ac3ac92 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6392b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/6ddbe8b7-db37-4ccb-b75e-7f428a525f42_isr_994640_33.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/6ddbe8b7-db37-4ccb-b75e-7f428a525f42_isr_994640_33.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App characterizeImage, waiting on task 750 parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 751 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App calibrate, waiting on task 751 parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 752 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash f20aa12ecc411ebbeb00dc8d923a4c75 parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6392840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/0822b7b1-2e3c-4e33-a42b-9e7942f6c426_isr_994638_13.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/0822b7b1-2e3c-4e33-a42b-9e7942f6c426_isr_994638_13.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App characterizeImage, waiting on task 753 parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 754 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App calibrate, waiting on task 754 parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 755 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash 6baeecfa8e818a795dfb8fcaba865455 parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6392520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0f68dd10-2266-4009-a51f-e8fa45777583_isr_994765_48.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0f68dd10-2266-4009-a51f-e8fa45777583_isr_994765_48.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App characterizeImage, waiting on task 756 parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 757 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App calibrate, waiting on task 757 parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash ea158f8c40cd65cc13f6e90a76cd6975 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6392200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4342d7cf-5df4-4459-8022-80675a168afd_isr_994766_27.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4342d7cf-5df4-4459-8022-80675a168afd_isr_994766_27.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App characterizeImage, waiting on task 759 parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 760 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App calibrate, waiting on task 760 parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 761 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash 9b62d5d58ba3e3f439dd63eb0a6a0843 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/71b0e97a-21c2-474e-80d9-38c421d2734c_isr_994765_61.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/71b0e97a-21c2-474e-80d9-38c421d2734c_isr_994765_61.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App characterizeImage, waiting on task 762 parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 763 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App calibrate, waiting on task 763 parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 764 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash fca5a72c07152e60b888a9569891f6f0 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63934c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:13:04 proc_lsst.multi:170 [INFO] [multi] job_id 17057292 proc_lsst.multi INFO: [multi] job_id 17057292 2024-03-15 04:13:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 04:13:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057292 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057292 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17057292 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-15 04:13:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-15 04:13:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 04:13:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/5b9c8767-fabb-443d-988c-c72202ec7be4_isr_994638_25.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/5b9c8767-fabb-443d-988c-c72202ec7be4_isr_994638_25.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App characterizeImage, waiting on task 765 parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 766 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App calibrate, waiting on task 766 parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 767 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash aca1adb99b46755008c8fc9ffb910af3 parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/28e867cf-1ac8-40b2-9684-e420957ae04a_isr_994766_22.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/28e867cf-1ac8-40b2-9684-e420957ae04a_isr_994766_22.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App characterizeImage, waiting on task 768 parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 769 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App calibrate, waiting on task 769 parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 770 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 8191b65d0f3f4b54d3a8eb8b461f0feb parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ec5e98b5-813b-41c7-9699-5a51299a1773_isr_994638_12.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ec5e98b5-813b-41c7-9699-5a51299a1773_isr_994638_12.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App characterizeImage, waiting on task 771 parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 772 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App calibrate, waiting on task 772 parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 773 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash ce1def2c1cbda62368127019ba2386aa parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/02dfd48f-2674-473f-be04-7c1e2b0d0bef_isr_994765_1.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/02dfd48f-2674-473f-be04-7c1e2b0d0bef_isr_994765_1.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 775 submitted for App characterizeImage, waiting on task 774 parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 775 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App calibrate, waiting on task 775 parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 776 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash 64f1e25f024e3f9723950d9e32fc594e parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f4996f73-618c-4dd3-9832-d5238b62ff1f_isr_994640_56.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f4996f73-618c-4dd3-9832-d5238b62ff1f_isr_994640_56.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App characterizeImage, waiting on task 777 parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 778 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App calibrate, waiting on task 778 parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 779 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 06d29f5b317e52f10ae4d88249927a9f parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/525b1d2d-cf87-4daf-b0aa-149b006f1866_isr_994766_41.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/525b1d2d-cf87-4daf-b0aa-149b006f1866_isr_994766_41.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App characterizeImage, waiting on task 780 parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App calibrate, waiting on task 781 parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 782 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash c6d8ef900bc2b01cfc635684c340bf22 parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c912e725-15e2-410f-ad42-64c64aa65d4c_isr_994640_28.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c912e725-15e2-410f-ad42-64c64aa65d4c_isr_994640_28.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App characterizeImage, waiting on task 783 parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 784 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App calibrate, waiting on task 784 parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 785 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash f710fe6b4d2d09ce45e62f986c667f61 parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bd828462-8a77-4638-8ac6-f05a4c859409_isr_994766_9.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bd828462-8a77-4638-8ac6-f05a4c859409_isr_994766_9.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App characterizeImage, waiting on task 786 parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 787 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App calibrate, waiting on task 787 parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 788 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash e7788d81533e571c135b35689ed0136f parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/5358044d-62e1-41de-8086-f43432d4e6f5_isr_994639_17.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/5358044d-62e1-41de-8086-f43432d4e6f5_isr_994639_17.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App characterizeImage, waiting on task 789 parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 790 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App calibrate, waiting on task 790 parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 791 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 3fada77e476be206cec41e2ca692509d parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dcae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4c8c5ccd-91a0-4979-98d2-5b48103c6225_isr_994640_55.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4c8c5ccd-91a0-4979-98d2-5b48103c6225_isr_994640_55.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App characterizeImage, waiting on task 792 parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 793 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App calibrate, waiting on task 793 parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 794 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 2af7935273c33699aad47ca8adb15fe4 parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b5670708-1919-4fa3-8a2b-49354f8918bd_isr_994766_6.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b5670708-1919-4fa3-8a2b-49354f8918bd_isr_994766_6.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App characterizeImage, waiting on task 795 parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 796 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App calibrate, waiting on task 796 parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 797 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash ae8b02c788a8ccb1a454c0e52d6be460 parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/cff76e4b-4b64-4be5-812d-2d25079d2ea9_isr_994638_11.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/cff76e4b-4b64-4be5-812d-2d25079d2ea9_isr_994638_11.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App characterizeImage, waiting on task 798 parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 799 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App calibrate, waiting on task 799 parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 800 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash 3a468127a98120be2cc68ef0846d4334 parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/e16a2c8a-5318-4656-88f5-b851a0b0831f_isr_994638_49.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/e16a2c8a-5318-4656-88f5-b851a0b0831f_isr_994638_49.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App characterizeImage, waiting on task 801 parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 802 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App calibrate, waiting on task 802 parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 803 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 79527440cf094cf167014b5cd2ac20e6 parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b92997b7-e554-4200-b33e-67422c668ddb_isr_994638_43.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b92997b7-e554-4200-b33e-67422c668ddb_isr_994638_43.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App characterizeImage, waiting on task 804 parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 805 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App calibrate, waiting on task 805 parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 806 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash e45b0034cf8c3380303309e4849de02a parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63ddc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/dde15828-2f8b-4998-afe2-2abd56fde916_isr_994767_7.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/dde15828-2f8b-4998-afe2-2abd56fde916_isr_994767_7.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App characterizeImage, waiting on task 807 parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 808 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App calibrate, waiting on task 808 parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 809 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash 0ad4e46a568b73aac5d15e535a2e1d01 parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63deb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/817b0797-232e-4302-b45d-89157a3af5e3_isr_994639_38.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/817b0797-232e-4302-b45d-89157a3af5e3_isr_994639_38.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App characterizeImage, waiting on task 810 parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 811 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App calibrate, waiting on task 811 parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 812 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash 42850edf15390d8f0842dcf48c438757 parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63de840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6a21db39-074b-4956-8a4c-6efbc5fc534b_isr_994638_52.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6a21db39-074b-4956-8a4c-6efbc5fc534b_isr_994638_52.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App characterizeImage, waiting on task 813 parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 814 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App calibrate, waiting on task 814 parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 815 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 1846dbeb6e0b64496e6d5b0bf860b536 parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63de520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6e877c52-ceeb-4d89-91d3-b8fe0a2611b6_isr_994639_10.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6e877c52-ceeb-4d89-91d3-b8fe0a2611b6_isr_994639_10.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App characterizeImage, waiting on task 816 parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 817 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App calibrate, waiting on task 817 parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 818 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash bbfa43bf6114be44fe342b6a4c6dbad5 parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63de200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d3a03c5e-1b49-4a11-8d16-a963b7aced5d_isr_994638_32.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d3a03c5e-1b49-4a11-8d16-a963b7aced5d_isr_994638_32.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App characterizeImage, waiting on task 819 parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 820 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App calibrate, waiting on task 820 parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 821 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash f4e1cd1a73f4294062621076538996d9 parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63ddee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/4eda5726-9ad4-4789-914e-8ee54ca891c4_isr_994767_38.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/4eda5726-9ad4-4789-914e-8ee54ca891c4_isr_994767_38.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App characterizeImage, waiting on task 822 parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 823 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App calibrate, waiting on task 823 parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 824 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash fe478731c25f4a6b2b7d68ce56f7cd6d parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63df380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:13:04 proc_lsst.multi:170 [INFO] [multi] job_id 17057293 proc_lsst.multi INFO: [multi] job_id 17057293 2024-03-15 04:13:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 04:13:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057293 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057293 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17057293 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-15 04:13:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-15 04:13:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 04:13:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d6e1ef57-09d6-47c8-aa31-f52092837faa_isr_994766_44.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d6e1ef57-09d6-47c8-aa31-f52092837faa_isr_994766_44.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App characterizeImage, waiting on task 825 parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 826 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App calibrate, waiting on task 826 parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash 8490be1805e037ab66ca90f2bd0c792d parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63df420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/df4f2266-4250-4c33-a2a7-9178b8857451_isr_994766_34.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/df4f2266-4250-4c33-a2a7-9178b8857451_isr_994766_34.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App characterizeImage, waiting on task 828 parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 829 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App calibrate, waiting on task 829 parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 830 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash cff6588e0d96f4b987bde9b0e0943c6c parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63df7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4ff18f0f-5d82-4298-9e45-4b00b635c150_isr_994640_48.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4ff18f0f-5d82-4298-9e45-4b00b635c150_isr_994640_48.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App characterizeImage, waiting on task 831 parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 832 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App calibrate, waiting on task 832 parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 833 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash 2dd23e6d2b30dd261c6b185f429a23c7 parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/af55885e-f6db-40d1-8583-c9fba42f4f75_isr_994765_24.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/af55885e-f6db-40d1-8583-c9fba42f4f75_isr_994765_24.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App characterizeImage, waiting on task 834 parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 835 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App calibrate, waiting on task 835 parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 836 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash 05ad338881a73afa3fded65f00bc323f parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/12a7de2c-05e8-457c-b9ae-56b146925244_isr_994639_23.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/12a7de2c-05e8-457c-b9ae-56b146925244_isr_994639_23.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App characterizeImage, waiting on task 837 parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 838 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App calibrate, waiting on task 838 parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 839 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash 5941fdf04f06a70b811e4a4064c75259 parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b6db45a3-5c31-49de-94bb-30ec36b66f12_isr_994638_45.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b6db45a3-5c31-49de-94bb-30ec36b66f12_isr_994638_45.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App characterizeImage, waiting on task 840 parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 841 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App calibrate, waiting on task 841 parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 842 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash 63e48b4558479b293752ebf838b8ee24 parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/94cf4a76-b9b8-4f45-89a5-b7ad419573e3_isr_994639_58.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/94cf4a76-b9b8-4f45-89a5-b7ad419573e3_isr_994639_58.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App characterizeImage, waiting on task 843 parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 844 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App calibrate, waiting on task 844 parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 845 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash ed391d4dcee4043b215e62f0c1025315 parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/6e60b9dc-debe-446d-a2dd-dc1b39247ab0_isr_994766_62.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/6e60b9dc-debe-446d-a2dd-dc1b39247ab0_isr_994766_62.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App characterizeImage, waiting on task 846 parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 847 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App calibrate, waiting on task 847 parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 848 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash 37977e7397f49f8bb5d22d45cfc53821 parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/2d0706db-7600-4b0e-a8f6-638ed3bb35c0_isr_994767_5.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/2d0706db-7600-4b0e-a8f6-638ed3bb35c0_isr_994767_5.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App characterizeImage, waiting on task 849 parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App calibrate, waiting on task 850 parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 851 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash fea4a89d6aeb204b3b3322c1fd2d7fa4 parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cf5619b3-fad7-4952-9e74-5846dffd537a_isr_994639_35.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cf5619b3-fad7-4952-9e74-5846dffd537a_isr_994639_35.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App characterizeImage, waiting on task 852 parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 853 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App calibrate, waiting on task 853 parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 854 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash 37c63c98b37067feff3d9aa2fd2bc9b4 parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/12aed698-4e9b-40c7-a6a6-b90929512803_isr_994766_40.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/12aed698-4e9b-40c7-a6a6-b90929512803_isr_994766_40.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App characterizeImage, waiting on task 855 parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 856 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App calibrate, waiting on task 856 parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 857 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 00cea9b389115e2273cdaa628532deaf parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d282c0c5-b182-4cf0-9b36-fd74111ba68e_isr_994638_9.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d282c0c5-b182-4cf0-9b36-fd74111ba68e_isr_994638_9.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App characterizeImage, waiting on task 858 parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 859 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App calibrate, waiting on task 859 parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 860 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash b0d77ffce04810e6b019223f0864d2fe parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/6d54a996-ca86-4dd7-bf32-855a5ea4df71_isr_994640_7.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/6d54a996-ca86-4dd7-bf32-855a5ea4df71_isr_994640_7.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App characterizeImage, waiting on task 861 parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 862 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App calibrate, waiting on task 862 parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 863 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash fb3aa121c03ee57270812d9e25a19710 parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/283c171a-e1dc-4492-b1d5-485d7c3101ae_isr_994766_29.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/283c171a-e1dc-4492-b1d5-485d7c3101ae_isr_994766_29.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App characterizeImage, waiting on task 864 parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 865 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App calibrate, waiting on task 865 parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 866 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 45a7d0191598984214eb095c6b98e14f parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fc16a0c9-27cd-4186-8dab-391bd90151e0_isr_994638_23.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fc16a0c9-27cd-4186-8dab-391bd90151e0_isr_994638_23.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App characterizeImage, waiting on task 867 parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 868 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App calibrate, waiting on task 868 parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 869 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 3f8e499fa565d5e9152b87060e522619 parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8648817b-1f52-4add-85a1-905dd5a4af48_isr_994767_52.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8648817b-1f52-4add-85a1-905dd5a4af48_isr_994767_52.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App characterizeImage, waiting on task 870 parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 871 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App calibrate, waiting on task 871 parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 872 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash d86e08b70babaf7f6295a1b464f6a492 parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b11a8cde-8ac5-4af3-ba48-9705d9061e8c_isr_994766_1.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b11a8cde-8ac5-4af3-ba48-9705d9061e8c_isr_994766_1.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App characterizeImage, waiting on task 873 parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 874 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App calibrate, waiting on task 874 parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 875 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash 00b4af5246af24d84e2b28d4e5347369 parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/97f06f5c-b65c-45e9-8d9a-c948aa8366e8_isr_994638_19.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/97f06f5c-b65c-45e9-8d9a-c948aa8366e8_isr_994638_19.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App characterizeImage, waiting on task 876 parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 877 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App calibrate, waiting on task 877 parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 878 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash 70b5571b7ff954a850938af682035a2a parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0033d07b-63e9-4235-a55a-e45b08bf64b5_isr_994765_33.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0033d07b-63e9-4235-a55a-e45b08bf64b5_isr_994765_33.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App characterizeImage, waiting on task 879 parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 880 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App calibrate, waiting on task 880 parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 881 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash 18c820aff632fc602ec3774c74f5cde7 parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8ca8649a-4dce-4b2a-aa2d-3f9ddde65cf1_isr_994638_55.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8ca8649a-4dce-4b2a-aa2d-3f9ddde65cf1_isr_994638_55.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App characterizeImage, waiting on task 882 parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 883 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App calibrate, waiting on task 883 parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 884 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash de8df5b6b2e61fb5eb4d0100cbe9bcdd parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/95ee00da-fbf8-4009-8af3-e7a870ea9e00_isr_994638_57.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/95ee00da-fbf8-4009-8af3-e7a870ea9e00_isr_994638_57.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App characterizeImage, waiting on task 885 parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 886 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App calibrate, waiting on task 886 parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 887 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash c794c2bda0345b0776a3e1b9c7ea54d7 parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/050ac7da-cefa-41a5-8c3e-fe49618c194e_isr_994765_55.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/050ac7da-cefa-41a5-8c3e-fe49618c194e_isr_994765_55.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App characterizeImage, waiting on task 888 parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 889 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App calibrate, waiting on task 889 parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 890 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash 175150868c7758d8a722d40375f7402b parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9c889813-5068-4d7c-813e-5f6c75608271_isr_994767_15.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9c889813-5068-4d7c-813e-5f6c75608271_isr_994767_15.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App characterizeImage, waiting on task 891 parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 892 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App calibrate, waiting on task 892 parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 893 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash fcbe0c44fd8b82a736987b14d94e8aaf parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/3fdf753c-fd33-4166-91af-ae3d8804f338_isr_994639_30.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/3fdf753c-fd33-4166-91af-ae3d8804f338_isr_994639_30.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App characterizeImage, waiting on task 894 parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 895 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App calibrate, waiting on task 895 parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash 065bc8dc150acb9329daad91ba5e8d17 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/94919847-6f47-4664-bc76-dbff1649a336_isr_994765_15.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/94919847-6f47-4664-bc76-dbff1649a336_isr_994765_15.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App characterizeImage, waiting on task 897 parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 898 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App calibrate, waiting on task 898 parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 899 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash e5797cd8d5f3549cd720abcdc4937c42 parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/332cacf8-9326-4724-ab64-4160c0cd0d52_isr_994640_5.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/332cacf8-9326-4724-ab64-4160c0cd0d52_isr_994640_5.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App characterizeImage, waiting on task 900 parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 901 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App calibrate, waiting on task 901 parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 902 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash 10a47b71a8ee2b5044520dfc850959bc parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:13:04 proc_lsst.multi:170 [INFO] [multi] job_id 17057294 proc_lsst.multi INFO: [multi] job_id 17057294 2024-03-15 04:13:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 04:13:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057294 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057294 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17057294 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-15 04:13:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-15 04:13:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 04:13:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0209fa9a-3438-405a-835b-fdc0646bbbc0_isr_994765_58.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0209fa9a-3438-405a-835b-fdc0646bbbc0_isr_994765_58.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App characterizeImage, waiting on task 903 parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 904 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App calibrate, waiting on task 904 parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 905 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash 8bdce1c2a0ca7bbdf32b879dea0f9e75 parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62685e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/603939b0-3596-4516-a143-8694abe075cb_isr_994640_19.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/603939b0-3596-4516-a143-8694abe075cb_isr_994640_19.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App characterizeImage, waiting on task 906 parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 907 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App calibrate, waiting on task 907 parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 908 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash c3e060a24981d6b84929bbc0587353c6 parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62689a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/47b87710-95f3-4e73-a8a9-e850bc9f32f4_isr_994640_60.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/47b87710-95f3-4e73-a8a9-e850bc9f32f4_isr_994640_60.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App characterizeImage, waiting on task 909 parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 910 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App calibrate, waiting on task 910 parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 911 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash 9da51af4838194d77fd6753dfe279eaa parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/00b2bff6-2833-4464-a5a2-0de045e38394_isr_994767_13.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/00b2bff6-2833-4464-a5a2-0de045e38394_isr_994767_13.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App characterizeImage, waiting on task 912 parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 913 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App calibrate, waiting on task 913 parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 914 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash d6a463fa6e8d70960697cc5245209eb8 parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3fa92cd0-91b5-453d-bde0-6287b4644cbd_isr_994638_44.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3fa92cd0-91b5-453d-bde0-6287b4644cbd_isr_994638_44.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App characterizeImage, waiting on task 915 parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 916 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App calibrate, waiting on task 916 parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 917 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash d080e10f15dd48492225a27d70623574 parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/08aeec8a-8509-4590-b1c8-3effe2d0da3b_isr_994640_42.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/08aeec8a-8509-4590-b1c8-3effe2d0da3b_isr_994640_42.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App characterizeImage, waiting on task 918 parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App calibrate, waiting on task 919 parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 920 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash 327265bb71d3e6ff605b6f68a76dc36c parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/dad5dece-4e19-41dc-b7e9-baf57dbe17a9_isr_994765_36.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/dad5dece-4e19-41dc-b7e9-baf57dbe17a9_isr_994765_36.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App characterizeImage, waiting on task 921 parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 922 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App calibrate, waiting on task 922 parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 923 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash d0f0e5123967aa1949a6a8079288ebda parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5f1e4065-4a7e-466d-8c7b-9899729eb344_isr_994767_50.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5f1e4065-4a7e-466d-8c7b-9899729eb344_isr_994767_50.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App characterizeImage, waiting on task 924 parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 925 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App calibrate, waiting on task 925 parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 926 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash bcce452a3d8c6e8c48ad6ef6959ea498 parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5db7f05d-f601-4ba1-900f-da3403b851ba_isr_994767_33.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5db7f05d-f601-4ba1-900f-da3403b851ba_isr_994767_33.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App characterizeImage, waiting on task 927 parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 928 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App calibrate, waiting on task 928 parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 929 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 930 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 930 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 930 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 930 has memoization hash 6f3f87f3a8ee02f83664be6fbd7e0d78 parsl.dataflow.memoization INFO: Task 930 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a40e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 930 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9586f860-c742-479f-af80-fdd4b665f9be_isr_994765_46.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9586f860-c742-479f-af80-fdd4b665f9be_isr_994765_46.stderr parsl.dataflow.dflow DEBUG: Task 931 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 931 submitted for App characterizeImage, waiting on task 930 parsl.dataflow.dflow DEBUG: Task 931 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 931 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 932 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 932 submitted for App calibrate, waiting on task 931 parsl.dataflow.dflow DEBUG: Task 932 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 932 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 933 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 933 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 933 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 933 has memoization hash cfbb1b2db08446f230d5c6217a9bac73 parsl.dataflow.memoization INFO: Task 933 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 933 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/4d35f123-2774-4b0a-ba03-87650cb1967e_isr_994765_49.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/4d35f123-2774-4b0a-ba03-87650cb1967e_isr_994765_49.stderr parsl.dataflow.dflow DEBUG: Task 934 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 934 submitted for App characterizeImage, waiting on task 933 parsl.dataflow.dflow DEBUG: Task 934 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 934 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 935 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 935 submitted for App calibrate, waiting on task 934 parsl.dataflow.dflow DEBUG: Task 935 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 935 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 936 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 936 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 936 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 936 has memoization hash e527c4a11413f003f1a2db4863303e26 parsl.dataflow.memoization INFO: Task 936 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 936 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/cd04f394-037e-4369-947b-e461d1f5db6c_isr_994638_18.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/cd04f394-037e-4369-947b-e461d1f5db6c_isr_994638_18.stderr parsl.dataflow.dflow DEBUG: Task 937 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 937 submitted for App characterizeImage, waiting on task 936 parsl.dataflow.dflow DEBUG: Task 937 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 937 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 938 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 938 submitted for App calibrate, waiting on task 937 parsl.dataflow.dflow DEBUG: Task 938 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 938 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 939 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 939 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 939 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 939 has memoization hash e8eb5071789a9d7ff9209e6253e8db77 parsl.dataflow.memoization INFO: Task 939 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 939 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/491008f5-3895-4314-9356-701808f65637_isr_994639_55.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/491008f5-3895-4314-9356-701808f65637_isr_994639_55.stderr parsl.dataflow.dflow DEBUG: Task 940 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 940 submitted for App characterizeImage, waiting on task 939 parsl.dataflow.dflow DEBUG: Task 940 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 940 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 941 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 941 submitted for App calibrate, waiting on task 940 parsl.dataflow.dflow DEBUG: Task 941 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 941 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 942 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 942 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 942 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 942 has memoization hash 6e0af4e798cd301f287f3e26c14383af parsl.dataflow.memoization INFO: Task 942 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 942 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/3cc04a7e-df09-4d80-886c-2165d3199add_isr_994767_3.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/3cc04a7e-df09-4d80-886c-2165d3199add_isr_994767_3.stderr parsl.dataflow.dflow DEBUG: Task 943 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 943 submitted for App characterizeImage, waiting on task 942 parsl.dataflow.dflow DEBUG: Task 943 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 943 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 944 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 944 submitted for App calibrate, waiting on task 943 parsl.dataflow.dflow DEBUG: Task 944 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 944 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 945 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 945 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 945 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 945 has memoization hash 89b59ee006f58b61e0170ef052a1d56d parsl.dataflow.memoization INFO: Task 945 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 945 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/0335de7b-4b43-4459-9be7-955713cebc28_isr_994766_28.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/0335de7b-4b43-4459-9be7-955713cebc28_isr_994766_28.stderr parsl.dataflow.dflow DEBUG: Task 946 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 946 submitted for App characterizeImage, waiting on task 945 parsl.dataflow.dflow DEBUG: Task 946 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 946 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 947 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 947 submitted for App calibrate, waiting on task 946 parsl.dataflow.dflow DEBUG: Task 947 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 947 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 948 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 948 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 948 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 948 has memoization hash 4cf412c958e1a2d05204dd5f98604d51 parsl.dataflow.memoization INFO: Task 948 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 948 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ee1c37b0-657a-459c-ba97-f292f87821e3_isr_994767_59.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ee1c37b0-657a-459c-ba97-f292f87821e3_isr_994767_59.stderr parsl.dataflow.dflow DEBUG: Task 949 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 949 submitted for App characterizeImage, waiting on task 948 parsl.dataflow.dflow DEBUG: Task 949 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 949 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 950 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 950 submitted for App calibrate, waiting on task 949 parsl.dataflow.dflow DEBUG: Task 950 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 950 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 951 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 951 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 951 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 951 has memoization hash 06cf90d161523d42ea869c29124a3f9d parsl.dataflow.memoization INFO: Task 951 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 951 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d9795d6d-adeb-4f36-84cb-350d64f1c51b_isr_994639_25.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d9795d6d-adeb-4f36-84cb-350d64f1c51b_isr_994639_25.stderr parsl.dataflow.dflow DEBUG: Task 952 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 952 submitted for App characterizeImage, waiting on task 951 parsl.dataflow.dflow DEBUG: Task 952 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 952 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 953 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 953 submitted for App calibrate, waiting on task 952 parsl.dataflow.dflow DEBUG: Task 953 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 953 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 954 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 954 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 954 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 954 has memoization hash 177ec1df4c5c0cb56ecb2a9b163591db parsl.dataflow.memoization INFO: Task 954 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 954 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fb887205-43b8-400b-b464-bb7269b65e6e_isr_994765_21.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fb887205-43b8-400b-b464-bb7269b65e6e_isr_994765_21.stderr parsl.dataflow.dflow DEBUG: Task 955 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 955 submitted for App characterizeImage, waiting on task 954 parsl.dataflow.dflow DEBUG: Task 955 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 955 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 956 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 956 submitted for App calibrate, waiting on task 955 parsl.dataflow.dflow DEBUG: Task 956 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 956 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 957 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 957 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 957 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 957 has memoization hash d6d7bedca159df4c5beae427bf6c0399 parsl.dataflow.memoization INFO: Task 957 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 957 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/2d4512ef-f949-4358-aef4-5b023a32ec3c_isr_994638_41.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/2d4512ef-f949-4358-aef4-5b023a32ec3c_isr_994638_41.stderr parsl.dataflow.dflow DEBUG: Task 958 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 958 submitted for App characterizeImage, waiting on task 957 parsl.dataflow.dflow DEBUG: Task 958 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 958 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 959 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 959 submitted for App calibrate, waiting on task 958 parsl.dataflow.dflow DEBUG: Task 959 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 959 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 960 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 960 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 960 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 960 has memoization hash 235264a953436d782fb3dcbf3be89ccf parsl.dataflow.memoization INFO: Task 960 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 960 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3de77fe8-0bcd-4ea5-b4fb-f50fc144f9d1_isr_994640_38.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3de77fe8-0bcd-4ea5-b4fb-f50fc144f9d1_isr_994640_38.stderr parsl.dataflow.dflow DEBUG: Task 961 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 961 submitted for App characterizeImage, waiting on task 960 parsl.dataflow.dflow DEBUG: Task 961 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 961 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 962 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 962 submitted for App calibrate, waiting on task 961 parsl.dataflow.dflow DEBUG: Task 962 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 962 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 963 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 963 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 963 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 963 has memoization hash 5ac374fb6d6ae3c86e5dae6633c9c6c1 parsl.dataflow.memoization INFO: Task 963 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 963 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/66e9cbe2-3aa9-4f6d-9d84-603d26de4b8a_isr_994766_45.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/66e9cbe2-3aa9-4f6d-9d84-603d26de4b8a_isr_994766_45.stderr parsl.dataflow.dflow DEBUG: Task 964 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 964 submitted for App characterizeImage, waiting on task 963 parsl.dataflow.dflow DEBUG: Task 964 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 964 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 965 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 965 submitted for App calibrate, waiting on task 964 parsl.dataflow.dflow DEBUG: Task 965 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 965 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 966 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 966 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 966 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 966 has memoization hash 867555e61398ae10e355e158146a7989 parsl.dataflow.memoization INFO: Task 966 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 966 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/9c328e1a-0ff1-4100-a171-bd0bfa5afef5_isr_994639_43.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/9c328e1a-0ff1-4100-a171-bd0bfa5afef5_isr_994639_43.stderr parsl.dataflow.dflow DEBUG: Task 967 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 967 submitted for App characterizeImage, waiting on task 966 parsl.dataflow.dflow DEBUG: Task 967 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 967 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 968 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 968 submitted for App calibrate, waiting on task 967 parsl.dataflow.dflow DEBUG: Task 968 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 968 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 969 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 969 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 969 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 969 has memoization hash e7556913dc59a19e8996e7b09c4cac97 parsl.dataflow.memoization INFO: Task 969 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 969 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b7d62dd4-875b-428a-80da-405a27bc92af_isr_994638_46.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b7d62dd4-875b-428a-80da-405a27bc92af_isr_994638_46.stderr parsl.dataflow.dflow DEBUG: Task 970 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 970 submitted for App characterizeImage, waiting on task 969 parsl.dataflow.dflow DEBUG: Task 970 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 970 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 971 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 971 submitted for App calibrate, waiting on task 970 parsl.dataflow.dflow DEBUG: Task 971 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 971 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 972 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 972 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 972 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 972 has memoization hash cbdbb865b36c6f255c451dcb08620472 parsl.dataflow.memoization INFO: Task 972 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 972 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/042cac36-6fd3-454d-8e69-7d0bf00b8cc7_isr_994767_6.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/042cac36-6fd3-454d-8e69-7d0bf00b8cc7_isr_994767_6.stderr parsl.dataflow.dflow DEBUG: Task 973 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 973 submitted for App characterizeImage, waiting on task 972 parsl.dataflow.dflow DEBUG: Task 973 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 973 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 974 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 974 submitted for App calibrate, waiting on task 973 parsl.dataflow.dflow DEBUG: Task 974 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 974 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 975 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 975 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 975 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 975 has memoization hash 416f0acb14554b256f8d49ea0a3eef48 parsl.dataflow.memoization INFO: Task 975 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 975 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/64c95a55-906d-453e-9fa8-05791965c528_isr_994767_48.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/64c95a55-906d-453e-9fa8-05791965c528_isr_994767_48.stderr parsl.dataflow.dflow DEBUG: Task 976 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 976 submitted for App characterizeImage, waiting on task 975 parsl.dataflow.dflow DEBUG: Task 976 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 976 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 977 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 977 submitted for App calibrate, waiting on task 976 parsl.dataflow.dflow DEBUG: Task 977 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 977 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 978 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 978 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 978 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 978 has memoization hash f47ded02f2bd184bcb0f58d6b40e3e35 parsl.dataflow.memoization INFO: Task 978 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 978 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/99e8a1ee-a647-4ff8-b6cc-4f78246f3757_isr_994766_8.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/99e8a1ee-a647-4ff8-b6cc-4f78246f3757_isr_994766_8.stderr parsl.dataflow.dflow DEBUG: Task 979 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 979 submitted for App characterizeImage, waiting on task 978 parsl.dataflow.dflow DEBUG: Task 979 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 979 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 980 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 980 submitted for App calibrate, waiting on task 979 parsl.dataflow.dflow DEBUG: Task 980 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 980 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 981 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 981 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 981 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 981 has memoization hash f706aecdad3086d910633609906c7c18 parsl.dataflow.memoization INFO: Task 981 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:13:05 proc_lsst.multi:170 [INFO] [multi] job_id 17057295 proc_lsst.multi INFO: [multi] job_id 17057295 2024-03-15 04:13:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 04:13:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057295 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057295 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17057295 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-15 04:13:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-15 04:13:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 04:13:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 981 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ad8607be-643f-494a-a0f4-cb87aeb0bcc7_isr_994766_24.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ad8607be-643f-494a-a0f4-cb87aeb0bcc7_isr_994766_24.stderr parsl.dataflow.dflow DEBUG: Task 982 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 982 submitted for App characterizeImage, waiting on task 981 parsl.dataflow.dflow DEBUG: Task 982 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 982 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 983 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 983 submitted for App calibrate, waiting on task 982 parsl.dataflow.dflow DEBUG: Task 983 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 983 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 984 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 984 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 984 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 984 has memoization hash 2282474ce88e2e52d2e5a038d7eec45e parsl.dataflow.memoization INFO: Task 984 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 984 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4ab5b021-4533-4a13-a5ad-69b498e1c69a_isr_994640_11.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4ab5b021-4533-4a13-a5ad-69b498e1c69a_isr_994640_11.stderr parsl.dataflow.dflow DEBUG: Task 985 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 985 submitted for App characterizeImage, waiting on task 984 parsl.dataflow.dflow DEBUG: Task 985 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 985 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 986 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 986 submitted for App calibrate, waiting on task 985 parsl.dataflow.dflow DEBUG: Task 986 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 986 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 987 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 987 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 987 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 987 has memoization hash dafa6ffaecd7b47eed27815afb233356 parsl.dataflow.memoization INFO: Task 987 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 987 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/28f13302-e0a4-4933-8179-1585df93ada4_isr_994638_26.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/28f13302-e0a4-4933-8179-1585df93ada4_isr_994638_26.stderr parsl.dataflow.dflow DEBUG: Task 988 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 988 submitted for App characterizeImage, waiting on task 987 parsl.dataflow.dflow DEBUG: Task 988 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 988 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 989 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 989 submitted for App calibrate, waiting on task 988 parsl.dataflow.dflow DEBUG: Task 989 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 989 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 990 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 990 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 990 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 990 has memoization hash f78602087c3e2a4852e4af040194ab29 parsl.dataflow.memoization INFO: Task 990 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 990 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87ca0c1f-94e1-4446-aee8-f00084209cf4_isr_994765_47.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87ca0c1f-94e1-4446-aee8-f00084209cf4_isr_994765_47.stderr parsl.dataflow.dflow DEBUG: Task 991 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 991 submitted for App characterizeImage, waiting on task 990 parsl.dataflow.dflow DEBUG: Task 991 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 991 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 992 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 992 submitted for App calibrate, waiting on task 991 parsl.dataflow.dflow DEBUG: Task 992 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 992 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 993 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 993 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 993 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 993 has memoization hash 5f90b94aa611c3fbdce4cf665b803f43 parsl.dataflow.memoization INFO: Task 993 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ec2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 993 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8e3b0c04-36df-4f5f-ada7-4dd5019ff437_isr_994638_51.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8e3b0c04-36df-4f5f-ada7-4dd5019ff437_isr_994638_51.stderr parsl.dataflow.dflow DEBUG: Task 994 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 994 submitted for App characterizeImage, waiting on task 993 parsl.dataflow.dflow DEBUG: Task 994 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 994 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 995 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 995 submitted for App calibrate, waiting on task 994 parsl.dataflow.dflow DEBUG: Task 995 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 995 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 996 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 996 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 996 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 996 has memoization hash b37c5001e2fa9411e547a2a20108171d parsl.dataflow.memoization INFO: Task 996 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ec5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 996 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/00169d10-aadc-4dce-92df-dea1658700e4_isr_994765_43.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/00169d10-aadc-4dce-92df-dea1658700e4_isr_994765_43.stderr parsl.dataflow.dflow DEBUG: Task 997 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 997 submitted for App characterizeImage, waiting on task 996 parsl.dataflow.dflow DEBUG: Task 997 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 997 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 998 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 998 submitted for App calibrate, waiting on task 997 parsl.dataflow.dflow DEBUG: Task 998 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 998 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 999 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 999 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 999 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 999 has memoization hash 24af49924a85098c3715717c8b2da6ed parsl.dataflow.memoization INFO: Task 999 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 999 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/0df11c18-734c-4155-8522-03527ec50ca9_isr_994640_10.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/0df11c18-734c-4155-8522-03527ec50ca9_isr_994640_10.stderr parsl.dataflow.dflow DEBUG: Task 1000 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1000 submitted for App characterizeImage, waiting on task 999 parsl.dataflow.dflow DEBUG: Task 1000 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1000 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1001 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1001 submitted for App calibrate, waiting on task 1000 parsl.dataflow.dflow DEBUG: Task 1001 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1001 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1002 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1002 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1002 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1002 has memoization hash 3491d5e8cc9ed8633baf515bb95524e6 parsl.dataflow.memoization INFO: Task 1002 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1002 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/affd8635-f64d-455b-8ff7-a42cb34f1c52_isr_994639_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/affd8635-f64d-455b-8ff7-a42cb34f1c52_isr_994639_13.stderr parsl.dataflow.dflow DEBUG: Task 1003 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1003 submitted for App characterizeImage, waiting on task 1002 parsl.dataflow.dflow DEBUG: Task 1003 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1003 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1004 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1004 submitted for App calibrate, waiting on task 1003 parsl.dataflow.dflow DEBUG: Task 1004 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1004 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1005 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1005 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1005 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1005 has memoization hash 750c52fc8e5b1b0575cdb07fbf76227d parsl.dataflow.memoization INFO: Task 1005 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1005 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bfb0dedf-f671-4485-be64-4a50a7da7b00_isr_994766_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bfb0dedf-f671-4485-be64-4a50a7da7b00_isr_994766_42.stderr parsl.dataflow.dflow DEBUG: Task 1006 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1006 submitted for App characterizeImage, waiting on task 1005 parsl.dataflow.dflow DEBUG: Task 1006 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1006 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1007 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1007 submitted for App calibrate, waiting on task 1006 parsl.dataflow.dflow DEBUG: Task 1007 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1007 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1008 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1008 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1008 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1008 has memoization hash 75e574a34d5930a2f3b698b6ac2e0f21 parsl.dataflow.memoization INFO: Task 1008 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1008 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/e23b2653-75a0-43c7-ab2a-e6792599d8ef_isr_994765_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/e23b2653-75a0-43c7-ab2a-e6792599d8ef_isr_994765_12.stderr parsl.dataflow.dflow DEBUG: Task 1009 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1009 submitted for App characterizeImage, waiting on task 1008 parsl.dataflow.dflow DEBUG: Task 1009 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1009 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1010 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1010 submitted for App calibrate, waiting on task 1009 parsl.dataflow.dflow DEBUG: Task 1010 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1010 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1011 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1011 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1011 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1011 has memoization hash 6240333f319e974f1a0bec30673c0875 parsl.dataflow.memoization INFO: Task 1011 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1011 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/16c640bd-930b-4ef3-b49e-e6a5daca1fc9_isr_994639_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/16c640bd-930b-4ef3-b49e-e6a5daca1fc9_isr_994639_4.stderr parsl.dataflow.dflow DEBUG: Task 1012 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1012 submitted for App characterizeImage, waiting on task 1011 parsl.dataflow.dflow DEBUG: Task 1012 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1012 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1013 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1013 submitted for App calibrate, waiting on task 1012 parsl.dataflow.dflow DEBUG: Task 1013 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1013 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1014 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1014 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1014 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1014 has memoization hash 8298c61f4d2f02814c5ab560f010b996 parsl.dataflow.memoization INFO: Task 1014 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1014 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/fb3c1144-fd99-4eea-bd98-7bcd70c1ffe4_isr_994767_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/fb3c1144-fd99-4eea-bd98-7bcd70c1ffe4_isr_994767_56.stderr parsl.dataflow.dflow DEBUG: Task 1015 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1015 submitted for App characterizeImage, waiting on task 1014 parsl.dataflow.dflow DEBUG: Task 1015 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1015 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1016 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1016 submitted for App calibrate, waiting on task 1015 parsl.dataflow.dflow DEBUG: Task 1016 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1016 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1017 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1017 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1017 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1017 has memoization hash e59865e4e8846d16822036f814c2796c parsl.dataflow.memoization INFO: Task 1017 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a77e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1017 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/c213fe68-637d-40dc-85fa-663af5f13067_isr_994639_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/c213fe68-637d-40dc-85fa-663af5f13067_isr_994639_29.stderr parsl.dataflow.dflow DEBUG: Task 1018 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1018 submitted for App characterizeImage, waiting on task 1017 parsl.dataflow.dflow DEBUG: Task 1018 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1018 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1019 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1019 submitted for App calibrate, waiting on task 1018 parsl.dataflow.dflow DEBUG: Task 1019 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1019 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1020 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1020 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1020 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1020 has memoization hash 114278151bd8453f44b92dc16b379024 parsl.dataflow.memoization INFO: Task 1020 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1020 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9fd801dd-ad2f-4327-aced-0f2d80bbce1a_isr_994765_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9fd801dd-ad2f-4327-aced-0f2d80bbce1a_isr_994765_17.stderr parsl.dataflow.dflow DEBUG: Task 1021 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1021 submitted for App characterizeImage, waiting on task 1020 parsl.dataflow.dflow DEBUG: Task 1021 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1021 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1022 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1022 submitted for App calibrate, waiting on task 1021 parsl.dataflow.dflow DEBUG: Task 1022 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1022 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1023 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1023 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1023 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1023 has memoization hash a62f5cf76033efc3da2738086d7331e7 parsl.dataflow.memoization INFO: Task 1023 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ec4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1023 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ebd8f983-5a76-4eb5-b660-7d762ac18197_isr_994766_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ebd8f983-5a76-4eb5-b660-7d762ac18197_isr_994766_4.stderr parsl.dataflow.dflow DEBUG: Task 1024 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1024 submitted for App characterizeImage, waiting on task 1023 parsl.dataflow.dflow DEBUG: Task 1024 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1024 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1025 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1025 submitted for App calibrate, waiting on task 1024 parsl.dataflow.dflow DEBUG: Task 1025 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1025 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1026 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1026 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1026 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1026 has memoization hash f9dc9aa8376e0d44dc910ab3ad8c413a parsl.dataflow.memoization INFO: Task 1026 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ec680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1026 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/3dd89a7b-d5f7-4df4-bcc3-621b7a0337d0_isr_994765_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/3dd89a7b-d5f7-4df4-bcc3-621b7a0337d0_isr_994765_8.stderr parsl.dataflow.dflow DEBUG: Task 1027 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1027 submitted for App characterizeImage, waiting on task 1026 parsl.dataflow.dflow DEBUG: Task 1027 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1027 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1028 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1028 submitted for App calibrate, waiting on task 1027 parsl.dataflow.dflow DEBUG: Task 1028 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1028 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1029 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1029 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1029 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1029 has memoization hash 1f991c10b5b5e51053bff342cce6b9a0 parsl.dataflow.memoization INFO: Task 1029 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62eca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1029 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/0d8129f6-6696-48b0-b43b-9c42bec4c201_isr_994766_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/0d8129f6-6696-48b0-b43b-9c42bec4c201_isr_994766_46.stderr parsl.dataflow.dflow DEBUG: Task 1030 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1030 submitted for App characterizeImage, waiting on task 1029 parsl.dataflow.dflow DEBUG: Task 1030 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1030 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1031 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1031 submitted for App calibrate, waiting on task 1030 parsl.dataflow.dflow DEBUG: Task 1031 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1031 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1032 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1032 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1032 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1032 has memoization hash 05c353f282724f9f2c1ff00fe51ebf51 parsl.dataflow.memoization INFO: Task 1032 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ecd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1032 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d10b7fa8-b697-4556-924c-e96bad362ca9_isr_994767_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d10b7fa8-b697-4556-924c-e96bad362ca9_isr_994767_22.stderr parsl.dataflow.dflow DEBUG: Task 1033 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1033 submitted for App characterizeImage, waiting on task 1032 parsl.dataflow.dflow DEBUG: Task 1033 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1033 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1034 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1034 submitted for App calibrate, waiting on task 1033 parsl.dataflow.dflow DEBUG: Task 1034 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1034 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1035 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1035 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1035 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1035 has memoization hash 418e886236a51fbf47cc1dd600390cbb parsl.dataflow.memoization INFO: Task 1035 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ed080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1035 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/dc7efac0-0ff2-4881-9637-bace858b5b50_isr_994639_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/dc7efac0-0ff2-4881-9637-bace858b5b50_isr_994639_19.stderr parsl.dataflow.dflow DEBUG: Task 1036 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1036 submitted for App characterizeImage, waiting on task 1035 parsl.dataflow.dflow DEBUG: Task 1036 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1036 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1037 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1037 submitted for App calibrate, waiting on task 1036 parsl.dataflow.dflow DEBUG: Task 1037 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1037 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1038 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1038 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1038 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1038 has memoization hash a75a4414c11e8d0c98ab04d50e72b6d8 parsl.dataflow.memoization INFO: Task 1038 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ed3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1038 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a0c40475-cac8-4838-aa97-8265cde4ddaa_isr_994765_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a0c40475-cac8-4838-aa97-8265cde4ddaa_isr_994765_29.stderr parsl.dataflow.dflow DEBUG: Task 1039 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1039 submitted for App characterizeImage, waiting on task 1038 parsl.dataflow.dflow DEBUG: Task 1039 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1039 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1040 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1040 submitted for App calibrate, waiting on task 1039 parsl.dataflow.dflow DEBUG: Task 1040 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1040 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1041 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1041 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1041 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1041 has memoization hash 4cbe903c4ca27c8c26a81a5fb0b25da3 parsl.dataflow.memoization INFO: Task 1041 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ed6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1041 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b69a1df8-aed9-4714-a750-7e2768f93ec8_isr_994765_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b69a1df8-aed9-4714-a750-7e2768f93ec8_isr_994765_13.stderr parsl.dataflow.dflow DEBUG: Task 1042 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1042 submitted for App characterizeImage, waiting on task 1041 parsl.dataflow.dflow DEBUG: Task 1042 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1042 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1043 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1043 submitted for App calibrate, waiting on task 1042 parsl.dataflow.dflow DEBUG: Task 1043 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1043 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1044 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1044 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1044 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1044 has memoization hash 774e18f253794470e1ed2f971eae161f parsl.dataflow.memoization INFO: Task 1044 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ed9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1044 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fc94ce19-3dc1-4501-8b82-2e8bfe7f4f53_isr_994765_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fc94ce19-3dc1-4501-8b82-2e8bfe7f4f53_isr_994765_30.stderr parsl.dataflow.dflow DEBUG: Task 1045 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1045 submitted for App characterizeImage, waiting on task 1044 parsl.dataflow.dflow DEBUG: Task 1045 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1045 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1046 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1046 submitted for App calibrate, waiting on task 1045 parsl.dataflow.dflow DEBUG: Task 1046 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1046 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1047 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1047 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1047 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1047 has memoization hash 97757d58dcefee3acdc77d44d0ea2dc6 parsl.dataflow.memoization INFO: Task 1047 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62edd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1047 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/dd02bcd2-298e-4294-abc6-562f2d026ac3_isr_994765_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/dd02bcd2-298e-4294-abc6-562f2d026ac3_isr_994765_56.stderr parsl.dataflow.dflow DEBUG: Task 1048 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1048 submitted for App characterizeImage, waiting on task 1047 parsl.dataflow.dflow DEBUG: Task 1048 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1048 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1049 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1049 submitted for App calibrate, waiting on task 1048 parsl.dataflow.dflow DEBUG: Task 1049 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1049 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1050 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1050 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1050 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1050 has memoization hash d3c5d5c929b39cb97ee492ec23edd1fe parsl.dataflow.memoization INFO: Task 1050 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ee020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1050 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/3f6beff2-9718-49f6-84fc-883386ea47dc_isr_994639_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/3f6beff2-9718-49f6-84fc-883386ea47dc_isr_994639_61.stderr parsl.dataflow.dflow DEBUG: Task 1051 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1051 submitted for App characterizeImage, waiting on task 1050 parsl.dataflow.dflow DEBUG: Task 1051 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1051 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1052 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1052 submitted for App calibrate, waiting on task 1051 parsl.dataflow.dflow DEBUG: Task 1052 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1052 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1053 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1053 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1053 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1053 has memoization hash eee5629d5f927b69ac4b5becc833d166 parsl.dataflow.memoization INFO: Task 1053 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ee340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1053 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f83f46e5-6a9f-4d86-a754-981911549b31_isr_994638_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f83f46e5-6a9f-4d86-a754-981911549b31_isr_994638_61.stderr parsl.dataflow.dflow DEBUG: Task 1054 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1054 submitted for App characterizeImage, waiting on task 1053 parsl.dataflow.dflow DEBUG: Task 1054 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1054 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1055 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1055 submitted for App calibrate, waiting on task 1054 parsl.dataflow.dflow DEBUG: Task 1055 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1055 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1056 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1056 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1056 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1056 has memoization hash 3a00cfd8f1ed912959c497f24883d461 parsl.dataflow.memoization INFO: Task 1056 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ee660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1056 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/96e57bd6-cf09-4eb5-b368-8420306649d5_isr_994639_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/96e57bd6-cf09-4eb5-b368-8420306649d5_isr_994639_42.stderr parsl.dataflow.dflow DEBUG: Task 1057 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1057 submitted for App characterizeImage, waiting on task 1056 parsl.dataflow.dflow DEBUG: Task 1057 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1057 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1058 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1058 submitted for App calibrate, waiting on task 1057 parsl.dataflow.dflow DEBUG: Task 1058 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1058 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1059 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1059 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1059 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1059 has memoization hash 7f2caa49b4658f0ee71f950aaf585b99 parsl.dataflow.memoization INFO: Task 1059 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ee980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1059 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/98f33088-cfd9-401f-831d-c9cd4342ae90_isr_994767_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/98f33088-cfd9-401f-831d-c9cd4342ae90_isr_994767_12.stderr parsl.dataflow.dflow DEBUG: Task 1060 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1060 submitted for App characterizeImage, waiting on task 1059 parsl.dataflow.dflow DEBUG: Task 1060 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1060 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1061 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1061 submitted for App calibrate, waiting on task 1060 parsl.dataflow.dflow DEBUG: Task 1061 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1061 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1062 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1062 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1062 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1062 has memoization hash d68c14b589d61a6a3621440cdc232808 parsl.dataflow.memoization INFO: Task 1062 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62eeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:13:05 proc_lsst.multi:170 [INFO] [multi] job_id 17057296 proc_lsst.multi INFO: [multi] job_id 17057296 2024-03-15 04:13:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 04:13:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057296 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057296 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17057296 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-15 04:13:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-15 04:13:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 04:13:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 1062 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/84a0b3fb-898d-43ca-bb02-8b8a0c3a693c_isr_994638_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/84a0b3fb-898d-43ca-bb02-8b8a0c3a693c_isr_994638_35.stderr parsl.dataflow.dflow DEBUG: Task 1063 will be sent to executor multi parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1063 submitted for App characterizeImage, waiting on task 1062 parsl.dataflow.dflow DEBUG: Task 1063 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1063 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1064 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1064 submitted for App calibrate, waiting on task 1063 parsl.dataflow.dflow DEBUG: Task 1064 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1064 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1065 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1065 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1065 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1065 has memoization hash 1b0768d26dd8be298677211ef8574479 parsl.dataflow.memoization INFO: Task 1065 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1065 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/30a9316c-0065-428e-b49a-a1a8e0bd2333_isr_994638_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/30a9316c-0065-428e-b49a-a1a8e0bd2333_isr_994638_37.stderr parsl.dataflow.dflow DEBUG: Task 1066 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1066 submitted for App characterizeImage, waiting on task 1065 parsl.dataflow.dflow DEBUG: Task 1066 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1066 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1067 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1067 submitted for App calibrate, waiting on task 1066 parsl.dataflow.dflow DEBUG: Task 1067 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1067 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1068 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1068 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1068 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1068 has memoization hash e68d0443a0127f8b3701028eaaa0b0d3 parsl.dataflow.memoization INFO: Task 1068 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63459e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1068 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/f1e8aecd-aad3-4476-b17c-a2d43bcc93c6_isr_994767_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/f1e8aecd-aad3-4476-b17c-a2d43bcc93c6_isr_994767_44.stderr parsl.dataflow.dflow DEBUG: Task 1069 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1069 submitted for App characterizeImage, waiting on task 1068 parsl.dataflow.dflow DEBUG: Task 1069 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1069 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1070 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1070 submitted for App calibrate, waiting on task 1069 parsl.dataflow.dflow DEBUG: Task 1070 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1070 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1071 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1071 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1071 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1071 has memoization hash aa9c6f2823edd5483eaac59bf51f72c5 parsl.dataflow.memoization INFO: Task 1071 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ef240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1071 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/3a028f41-1582-4640-8b95-d69a60e12da7_isr_994767_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/3a028f41-1582-4640-8b95-d69a60e12da7_isr_994767_26.stderr parsl.dataflow.dflow DEBUG: Task 1072 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1072 submitted for App characterizeImage, waiting on task 1071 parsl.dataflow.dflow DEBUG: Task 1072 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1072 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1073 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1073 submitted for App calibrate, waiting on task 1072 parsl.dataflow.dflow DEBUG: Task 1073 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1073 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1074 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1074 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1074 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1074 has memoization hash 0843b366231cd1b78c043e90314ad672 parsl.dataflow.memoization INFO: Task 1074 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ef560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1074 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/964e4059-084d-488b-a8bc-6e7a4e3d433c_isr_994638_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/964e4059-084d-488b-a8bc-6e7a4e3d433c_isr_994638_38.stderr parsl.dataflow.dflow DEBUG: Task 1075 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1075 submitted for App characterizeImage, waiting on task 1074 parsl.dataflow.dflow DEBUG: Task 1075 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1075 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1076 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1076 submitted for App calibrate, waiting on task 1075 parsl.dataflow.dflow DEBUG: Task 1076 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1076 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1077 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1077 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1077 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1077 has memoization hash 101a8dafa70af3794f713f035c337187 parsl.dataflow.memoization INFO: Task 1077 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ef880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1077 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/51895857-607c-40a1-981c-896f655b5a9e_isr_994638_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/51895857-607c-40a1-981c-896f655b5a9e_isr_994638_62.stderr parsl.dataflow.dflow DEBUG: Task 1078 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1078 submitted for App characterizeImage, waiting on task 1077 parsl.dataflow.dflow DEBUG: Task 1078 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1078 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1079 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1079 submitted for App calibrate, waiting on task 1078 parsl.dataflow.dflow DEBUG: Task 1079 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1079 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1080 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1080 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1080 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1080 has memoization hash 09d34ff29d28deef792bc57c69789822 parsl.dataflow.memoization INFO: Task 1080 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62efa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1080 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8765558c-1fa9-483b-bd46-29d8f43185bf_isr_994766_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8765558c-1fa9-483b-bd46-29d8f43185bf_isr_994766_33.stderr parsl.dataflow.dflow DEBUG: Task 1081 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1081 submitted for App characterizeImage, waiting on task 1080 parsl.dataflow.dflow DEBUG: Task 1081 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1081 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1082 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1082 submitted for App calibrate, waiting on task 1081 parsl.dataflow.dflow DEBUG: Task 1082 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1082 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1083 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1083 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1083 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1083 has memoization hash 94ebb1094c70eecf36c10dbb57ff5a9c parsl.dataflow.memoization INFO: Task 1083 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62efd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1083 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/693a14ae-b533-417e-b99b-40a137489bc0_isr_994640_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/693a14ae-b533-417e-b99b-40a137489bc0_isr_994640_26.stderr parsl.dataflow.dflow DEBUG: Task 1084 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1084 submitted for App characterizeImage, waiting on task 1083 parsl.dataflow.dflow DEBUG: Task 1084 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1084 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1085 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1085 submitted for App calibrate, waiting on task 1084 parsl.dataflow.dflow DEBUG: Task 1085 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1085 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1086 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1086 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1086 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1086 has memoization hash a95567619ff7291b6b68743da4fab74d parsl.dataflow.memoization INFO: Task 1086 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63458a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1086 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d4a7ff74-c5d8-4cd3-bfa6-4befcfb368fd_isr_994638_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d4a7ff74-c5d8-4cd3-bfa6-4befcfb368fd_isr_994638_58.stderr parsl.dataflow.dflow DEBUG: Task 1087 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1087 submitted for App characterizeImage, waiting on task 1086 parsl.dataflow.dflow DEBUG: Task 1087 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1087 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1088 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1088 submitted for App calibrate, waiting on task 1087 parsl.dataflow.dflow DEBUG: Task 1088 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1088 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1089 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1089 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1089 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1089 has memoization hash 95eea96c099d43f66cb8572e60a14b53 parsl.dataflow.memoization INFO: Task 1089 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1089 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/87c0c29f-27fe-48a7-a2ae-a28b87d93271_isr_994640_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/87c0c29f-27fe-48a7-a2ae-a28b87d93271_isr_994640_12.stderr parsl.dataflow.dflow DEBUG: Task 1090 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1090 submitted for App characterizeImage, waiting on task 1089 parsl.dataflow.dflow DEBUG: Task 1090 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1090 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1091 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1091 submitted for App calibrate, waiting on task 1090 parsl.dataflow.dflow DEBUG: Task 1091 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1091 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1092 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1092 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1092 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1092 has memoization hash 13fc4f53f2e6f21f555c57b4e17b8212 parsl.dataflow.memoization INFO: Task 1092 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63444a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1092 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/19f91076-1ae6-4605-a728-bbf140d2d79e_isr_994639_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/19f91076-1ae6-4605-a728-bbf140d2d79e_isr_994639_49.stderr parsl.dataflow.dflow DEBUG: Task 1093 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1093 submitted for App characterizeImage, waiting on task 1092 parsl.dataflow.dflow DEBUG: Task 1093 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1093 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1094 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1094 submitted for App calibrate, waiting on task 1093 parsl.dataflow.dflow DEBUG: Task 1094 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1094 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1095 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1095 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1095 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1095 has memoization hash dbec4e44ad131196951cc41c85057e5a parsl.dataflow.memoization INFO: Task 1095 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63447c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1095 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/93328619-5ad8-40ea-9b01-709f1c33705d_isr_994765_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/93328619-5ad8-40ea-9b01-709f1c33705d_isr_994765_23.stderr parsl.dataflow.dflow DEBUG: Task 1096 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1096 submitted for App characterizeImage, waiting on task 1095 parsl.dataflow.dflow DEBUG: Task 1096 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1096 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1097 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1097 submitted for App calibrate, waiting on task 1096 parsl.dataflow.dflow DEBUG: Task 1097 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1097 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1098 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1098 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1098 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1098 has memoization hash d57ebab56dcfc9eadf7f86484b7c1a94 parsl.dataflow.memoization INFO: Task 1098 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1098 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/69402ed9-fc8a-4a2c-af8b-6684d3cfae58_isr_994640_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/69402ed9-fc8a-4a2c-af8b-6684d3cfae58_isr_994640_36.stderr parsl.dataflow.dflow DEBUG: Task 1099 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1099 submitted for App characterizeImage, waiting on task 1098 parsl.dataflow.dflow DEBUG: Task 1099 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1099 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1100 submitted for App calibrate, waiting on task 1099 parsl.dataflow.dflow DEBUG: Task 1100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1101 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1101 has memoization hash 5817bc9ee66c9fca161c5ee81809c5b3 parsl.dataflow.memoization INFO: Task 1101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1101 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7f0d204f-4fbc-4aa8-838d-2c30d6743a73_isr_994638_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7f0d204f-4fbc-4aa8-838d-2c30d6743a73_isr_994638_1.stderr parsl.dataflow.dflow DEBUG: Task 1102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1102 submitted for App characterizeImage, waiting on task 1101 parsl.dataflow.dflow DEBUG: Task 1102 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1102 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1103 submitted for App calibrate, waiting on task 1102 parsl.dataflow.dflow DEBUG: Task 1103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1104 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1104 has memoization hash a7315f67355755df3d7ce70759417691 parsl.dataflow.memoization INFO: Task 1104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1104 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f46c4725-ced0-4461-a6a2-f46ec7401492_isr_994638_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f46c4725-ced0-4461-a6a2-f46ec7401492_isr_994638_2.stderr parsl.dataflow.dflow DEBUG: Task 1105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1105 submitted for App characterizeImage, waiting on task 1104 parsl.dataflow.dflow DEBUG: Task 1105 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1105 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1106 submitted for App calibrate, waiting on task 1105 parsl.dataflow.dflow DEBUG: Task 1106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1107 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1107 has memoization hash 71c34819c0f2c809155401b0e2bdf6fe parsl.dataflow.memoization INFO: Task 1107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1107 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/be35aa9d-706b-4395-b092-c159c9b0eb0b_isr_994767_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/be35aa9d-706b-4395-b092-c159c9b0eb0b_isr_994767_45.stderr parsl.dataflow.dflow DEBUG: Task 1108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1108 submitted for App characterizeImage, waiting on task 1107 parsl.dataflow.dflow DEBUG: Task 1108 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1108 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1109 submitted for App calibrate, waiting on task 1108 parsl.dataflow.dflow DEBUG: Task 1109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1110 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1110 has memoization hash d0b9b6ddd6c9b7ff6a611fc1f1a3a111 parsl.dataflow.memoization INFO: Task 1110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1110 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/74a2414e-3798-4cee-a342-2c11463ff863_isr_994765_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/74a2414e-3798-4cee-a342-2c11463ff863_isr_994765_32.stderr parsl.dataflow.dflow DEBUG: Task 1111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1111 submitted for App characterizeImage, waiting on task 1110 parsl.dataflow.dflow DEBUG: Task 1111 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1111 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1112 submitted for App calibrate, waiting on task 1111 parsl.dataflow.dflow DEBUG: Task 1112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1113 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1113 has memoization hash ea08ba7b322dc9be0dd67ba5734db6d1 parsl.dataflow.memoization INFO: Task 1113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6346200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1113 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4531eb06-ab8f-4e4e-9e1c-9167ea9c2573_isr_994640_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4531eb06-ab8f-4e4e-9e1c-9167ea9c2573_isr_994640_47.stderr parsl.dataflow.dflow DEBUG: Task 1114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1114 submitted for App characterizeImage, waiting on task 1113 parsl.dataflow.dflow DEBUG: Task 1114 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1114 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 1115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1115 submitted for App calibrate, waiting on task 1114 parsl.dataflow.dflow DEBUG: Task 1115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1115 has outstanding dependencies, so launch_if_ready skipping 2024-03-15 04:13:05 proc_lsst.multi:170 [INFO] [multi] job_id 17057297 proc_lsst.multi INFO: [multi] job_id 17057297 2024-03-15 04:13:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 04:13:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057297 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057297 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17057297 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-15 04:13:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-15 04:13:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 04:13:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:05 proc_lsst.multi:170 [INFO] [multi] job_id 17057298 proc_lsst.multi INFO: [multi] job_id 17057298 2024-03-15 04:13:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 04:13:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057298 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057298 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17057298 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-15 04:13:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-15 04:13:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 04:13:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:06 proc_lsst.multi:170 [INFO] [multi] job_id 17057299 proc_lsst.multi INFO: [multi] job_id 17057299 2024-03-15 04:13:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 04:13:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057299 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057299 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17057299 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-15 04:13:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-15 04:13:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 04:13:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:06 proc_lsst.multi:170 [INFO] [multi] job_id 17057300 proc_lsst.multi INFO: [multi] job_id 17057300 2024-03-15 04:13:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 04:13:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057300 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057300 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17057300 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-15 04:13:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-15 04:13:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 04:13:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:06 proc_lsst.multi:170 [INFO] [multi] job_id 17057301 proc_lsst.multi INFO: [multi] job_id 17057301 2024-03-15 04:13:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 04:13:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057301 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057301 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17057301 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-15 04:13:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-15 04:13:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 04:13:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:07 proc_lsst.multi:170 [INFO] [multi] job_id 17057302 proc_lsst.multi INFO: [multi] job_id 17057302 2024-03-15 04:13:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 04:13:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057302 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057302 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17057302 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-15 04:13:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-15 04:13:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 04:13:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:07 proc_lsst.multi:170 [INFO] [multi] job_id 17057303 proc_lsst.multi INFO: [multi] job_id 17057303 2024-03-15 04:13:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 04:13:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057303 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057303 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17057303 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-15 04:13:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-15 04:13:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 04:13:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:08 proc_lsst.multi:170 [INFO] [multi] job_id 17057304 proc_lsst.multi INFO: [multi] job_id 17057304 2024-03-15 04:13:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 04:13:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057304 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057304 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17057304 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-15 04:13:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:08 proc_lsst.multi:170 [INFO] [multi] job_id 17057305 proc_lsst.multi INFO: [multi] job_id 17057305 2024-03-15 04:13:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 04:13:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057305 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057305 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17057305 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-15 04:13:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:08 proc_lsst.multi:170 [INFO] [multi] job_id 17057306 proc_lsst.multi INFO: [multi] job_id 17057306 2024-03-15 04:13:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057306 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057306 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17057306 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-15 04:13:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:08 proc_lsst.multi:170 [INFO] [multi] job_id 17057307 proc_lsst.multi INFO: [multi] job_id 17057307 2024-03-15 04:13:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 04:13:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057307 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057307 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17057307 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-15 04:13:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:08 proc_lsst.multi:170 [INFO] [multi] job_id 17057308 proc_lsst.multi INFO: [multi] job_id 17057308 2024-03-15 04:13:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 04:13:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057308 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057308 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17057308 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-15 04:13:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:08 proc_lsst.multi:170 [INFO] [multi] job_id 17057309 proc_lsst.multi INFO: [multi] job_id 17057309 2024-03-15 04:13:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 04:13:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057309 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057309 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17057309 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-15 04:13:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 04:13:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:09 proc_lsst.multi:170 [INFO] [multi] job_id 17057310 proc_lsst.multi INFO: [multi] job_id 17057310 2024-03-15 04:13:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 04:13:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057310 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057310 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17057310 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-15 04:13:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-15 04:13:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 04:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:09 proc_lsst.multi:170 [INFO] [multi] job_id 17057311 proc_lsst.multi INFO: [multi] job_id 17057311 2024-03-15 04:13:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 04:13:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057311 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057311 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17057311 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-15 04:13:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-15 04:13:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 04:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:09 proc_lsst.multi:170 [INFO] [multi] job_id 17057312 proc_lsst.multi INFO: [multi] job_id 17057312 2024-03-15 04:13:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 04:13:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057312 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057312 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17057312 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-15 04:13:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-15 04:13:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 04:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:09 proc_lsst.multi:170 [INFO] [multi] job_id 17057313 proc_lsst.multi INFO: [multi] job_id 17057313 2024-03-15 04:13:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 04:13:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057313 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057313 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17057313 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-15 04:13:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-15 04:13:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 04:13:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:10 proc_lsst.multi:170 [INFO] [multi] job_id 17057314 proc_lsst.multi INFO: [multi] job_id 17057314 2024-03-15 04:13:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 04:13:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057314 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057314 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17057314 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-15 04:13:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:10 proc_lsst.multi:170 [INFO] [multi] job_id 17057315 proc_lsst.multi INFO: [multi] job_id 17057315 2024-03-15 04:13:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 04:13:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057315 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057315 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17057315 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-15 04:13:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:10 proc_lsst.multi:170 [INFO] [multi] job_id 17057316 proc_lsst.multi INFO: [multi] job_id 17057316 2024-03-15 04:13:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17057316 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17057316 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17057316 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-15 04:13:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:10 proc_lsst.multi:170 [INFO] [multi] job_id 17057317 proc_lsst.multi INFO: [multi] job_id 17057317 2024-03-15 04:13:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17057317 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17057317 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17057317 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-15 04:13:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:10 proc_lsst.multi:170 [INFO] [multi] job_id 17057318 proc_lsst.multi INFO: [multi] job_id 17057318 2024-03-15 04:13:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 04:13:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17057318 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17057318 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17057318 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 33, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/32 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 33 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 60 more blocks parsl.executors.status_handling INFO: Scaling out by 60 blocks parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-15 04:13:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:10 proc_lsst.multi:170 [INFO] [multi] job_id 17057319 proc_lsst.multi INFO: [multi] job_id 17057319 2024-03-15 04:13:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 04:13:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17057319 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17057319 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17057319 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-15 04:13:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 04:13:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:11 proc_lsst.multi:170 [INFO] [multi] job_id 17057320 proc_lsst.multi INFO: [multi] job_id 17057320 2024-03-15 04:13:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 04:13:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:11 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17057320 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17057320 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17057320 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-15 04:13:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:11 proc_lsst.multi:170 [INFO] [multi] job_id 17057321 proc_lsst.multi INFO: [multi] job_id 17057321 2024-03-15 04:13:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:11 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17057321 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17057321 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17057321 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-15 04:13:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:11 proc_lsst.multi:170 [INFO] [multi] job_id 17057322 proc_lsst.multi INFO: [multi] job_id 17057322 2024-03-15 04:13:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057322 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057322 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17057322 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-15 04:13:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:11 proc_lsst.multi:170 [INFO] [multi] job_id 17057323 proc_lsst.multi INFO: [multi] job_id 17057323 2024-03-15 04:13:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 04:13:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057323 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057323 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17057323 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-15 04:13:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:11 proc_lsst.multi:170 [INFO] [multi] job_id 17057324 proc_lsst.multi INFO: [multi] job_id 17057324 2024-03-15 04:13:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 04:13:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057324 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057324 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17057324 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-15 04:13:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 04:13:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:12 proc_lsst.multi:170 [INFO] [multi] job_id 17057325 proc_lsst.multi INFO: [multi] job_id 17057325 2024-03-15 04:13:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 04:13:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057325 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057325 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17057325 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-15 04:13:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:12 proc_lsst.multi:170 [INFO] [multi] job_id 17057326 proc_lsst.multi INFO: [multi] job_id 17057326 2024-03-15 04:13:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057326 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057326 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17057326 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-15 04:13:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:12 proc_lsst.multi:170 [INFO] [multi] job_id 17057327 proc_lsst.multi INFO: [multi] job_id 17057327 2024-03-15 04:13:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 04:13:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057327 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057327 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17057327 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-15 04:13:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:12 proc_lsst.multi:170 [INFO] [multi] job_id 17057328 proc_lsst.multi INFO: [multi] job_id 17057328 2024-03-15 04:13:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 04:13:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057328 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057328 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17057328 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-15 04:13:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 04:13:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:13 proc_lsst.multi:170 [INFO] [multi] job_id 17057329 proc_lsst.multi INFO: [multi] job_id 17057329 2024-03-15 04:13:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 04:13:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057329 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057329 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17057329 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-15 04:13:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:13 proc_lsst.multi:170 [INFO] [multi] job_id 17057330 proc_lsst.multi INFO: [multi] job_id 17057330 2024-03-15 04:13:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 04:13:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057330 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057330 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17057330 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-15 04:13:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:13 proc_lsst.multi:170 [INFO] [multi] job_id 17057331 proc_lsst.multi INFO: [multi] job_id 17057331 2024-03-15 04:13:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 04:13:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057331 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057331 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17057331 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-15 04:13:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 04:13:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:14 proc_lsst.multi:170 [INFO] [multi] job_id 17057332 proc_lsst.multi INFO: [multi] job_id 17057332 2024-03-15 04:13:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 04:13:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057332 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057332 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17057332 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-15 04:13:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:14 proc_lsst.multi:170 [INFO] [multi] job_id 17057333 proc_lsst.multi INFO: [multi] job_id 17057333 2024-03-15 04:13:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 04:13:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057333 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057333 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17057333 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-15 04:13:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:14 proc_lsst.multi:170 [INFO] [multi] job_id 17057334 proc_lsst.multi INFO: [multi] job_id 17057334 2024-03-15 04:13:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 04:13:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057334 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057334 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17057334 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-15 04:13:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 04:13:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:15 proc_lsst.multi:170 [INFO] [multi] job_id 17057335 proc_lsst.multi INFO: [multi] job_id 17057335 2024-03-15 04:13:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 04:13:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057335 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057335 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17057335 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-15 04:13:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:15 proc_lsst.multi:170 [INFO] [multi] job_id 17057336 proc_lsst.multi INFO: [multi] job_id 17057336 2024-03-15 04:13:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 04:13:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057336 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057336 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17057336 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-15 04:13:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:15 proc_lsst.multi:170 [INFO] [multi] job_id 17057337 proc_lsst.multi INFO: [multi] job_id 17057337 2024-03-15 04:13:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 04:13:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057337 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057337 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17057337 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-15 04:13:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:15 proc_lsst.multi:170 [INFO] [multi] job_id 17057338 proc_lsst.multi INFO: [multi] job_id 17057338 2024-03-15 04:13:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 04:13:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057338 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057338 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17057338 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-15 04:13:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 04:13:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:16 proc_lsst.multi:170 [INFO] [multi] job_id 17057339 proc_lsst.multi INFO: [multi] job_id 17057339 2024-03-15 04:13:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 04:13:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057339 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057339 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17057339 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-15 04:13:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:16 proc_lsst.multi:170 [INFO] [multi] job_id 17057340 proc_lsst.multi INFO: [multi] job_id 17057340 2024-03-15 04:13:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 04:13:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057340 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057340 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17057340 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-15 04:13:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:16 proc_lsst.multi:170 [INFO] [multi] job_id 17057341 proc_lsst.multi INFO: [multi] job_id 17057341 2024-03-15 04:13:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17057341 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17057341 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17057341 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-15 04:13:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:16 proc_lsst.multi:170 [INFO] [multi] job_id 17057342 proc_lsst.multi INFO: [multi] job_id 17057342 2024-03-15 04:13:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:16 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057342 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057342 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17057342 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-15 04:13:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:16 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:17 proc_lsst.multi:170 [INFO] [multi] job_id 17057343 proc_lsst.multi INFO: [multi] job_id 17057343 2024-03-15 04:13:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 04:13:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057343 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057343 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17057343 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-15 04:13:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d03add13cc7892a5fc548dce6590113 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash eac0ad43831f78b4ef2a6222e422c6c2 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b85120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/316c3d75-338f-4fd2-b6ac-a0aa2d1afa67_characterizeImage_994765_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/316c3d75-338f-4fd2-b6ac-a0aa2d1afa67_characterizeImage_994765_53.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/01983423-b790-4d50-9042-5f33de67dd99_isr_994765_53.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/01983423-b790-4d50-9042-5f33de67dd99_isr_994765_53.stderr 2024-03-15 04:13:17 proc_lsst.multi:170 [INFO] [multi] job_id 17057344 proc_lsst.multi INFO: [multi] job_id 17057344 2024-03-15 04:13:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 04:13:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057344 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057344 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17057344 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-15 04:13:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:17 proc_lsst.multi:170 [INFO] [multi] job_id 17057345 proc_lsst.multi INFO: [multi] job_id 17057345 2024-03-15 04:13:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 04:13:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:17 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057345 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057345 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17057345 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-15 04:13:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:17 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-15 04:13:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b15ab1b57043d7a4cf47052602070ce with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 07ab4612d148501f926266b40047d6a4 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b84c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/01f51214-cdf8-466f-81bb-8f28c60fb8eb_characterizeImage_994638_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/01f51214-cdf8-466f-81bb-8f28c60fb8eb_characterizeImage_994638_3.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/e9fc1f26-3afe-4d30-b6fe-dc96ca8b164e_isr_994638_3.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/e9fc1f26-3afe-4d30-b6fe-dc96ca8b164e_isr_994638_3.stderr 2024-03-15 04:13:18 proc_lsst.multi:170 [INFO] [multi] job_id 17057346 proc_lsst.multi INFO: [multi] job_id 17057346 2024-03-15 04:13:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057346 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057346 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17057346 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-15 04:13:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:18 proc_lsst.multi:170 [INFO] [multi] job_id 17057347 proc_lsst.multi INFO: [multi] job_id 17057347 2024-03-15 04:13:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 04:13:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057347 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057347 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17057347 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-15 04:13:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:18 proc_lsst.multi:170 [INFO] [multi] job_id 17057348 proc_lsst.multi INFO: [multi] job_id 17057348 2024-03-15 04:13:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 04:13:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:18 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057348 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057348 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17057348 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-15 04:13:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:18 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-15 04:13:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:19 proc_lsst.multi:170 [INFO] [multi] job_id 17057349 proc_lsst.multi INFO: [multi] job_id 17057349 2024-03-15 04:13:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 04:13:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057349 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057349 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17057349 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-15 04:13:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:19 proc_lsst.multi:170 [INFO] [multi] job_id 17057350 proc_lsst.multi INFO: [multi] job_id 17057350 2024-03-15 04:13:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 04:13:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057350 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057350 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17057350 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-15 04:13:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:19 proc_lsst.multi:170 [INFO] [multi] job_id 17057351 proc_lsst.multi INFO: [multi] job_id 17057351 2024-03-15 04:13:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 04:13:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:19 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057351 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057351 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17057351 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-15 04:13:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:19 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-15 04:13:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:20 proc_lsst.multi:170 [INFO] [multi] job_id 17057352 proc_lsst.multi INFO: [multi] job_id 17057352 2024-03-15 04:13:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 04:13:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057352 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057352 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17057352 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-15 04:13:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:20 proc_lsst.multi:170 [INFO] [multi] job_id 17057353 proc_lsst.multi INFO: [multi] job_id 17057353 2024-03-15 04:13:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 04:13:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057353 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057353 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17057353 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-15 04:13:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:20 proc_lsst.multi:170 [INFO] [multi] job_id 17057354 proc_lsst.multi INFO: [multi] job_id 17057354 2024-03-15 04:13:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 04:13:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057354 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057354 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17057354 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-15 04:13:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:20 proc_lsst.multi:170 [INFO] [multi] job_id 17057355 proc_lsst.multi INFO: [multi] job_id 17057355 2024-03-15 04:13:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 04:13:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057355 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057355 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17057355 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-15 04:13:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-15 04:13:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:21 proc_lsst.multi:170 [INFO] [multi] job_id 17057356 proc_lsst.multi INFO: [multi] job_id 17057356 2024-03-15 04:13:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 04:13:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057356 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057356 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17057356 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-15 04:13:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:21 proc_lsst.multi:170 [INFO] [multi] job_id 17057357 proc_lsst.multi INFO: [multi] job_id 17057357 2024-03-15 04:13:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 04:13:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057357 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057357 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17057357 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-15 04:13:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:21 proc_lsst.multi:170 [INFO] [multi] job_id 17057358 proc_lsst.multi INFO: [multi] job_id 17057358 2024-03-15 04:13:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 04:13:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057358 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057358 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17057358 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-15 04:13:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:21 proc_lsst.multi:170 [INFO] [multi] job_id 17057359 proc_lsst.multi INFO: [multi] job_id 17057359 2024-03-15 04:13:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 04:13:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057359 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057359 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17057359 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-15 04:13:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:21 proc_lsst.multi:170 [INFO] [multi] job_id 17057360 proc_lsst.multi INFO: [multi] job_id 17057360 2024-03-15 04:13:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 04:13:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057360 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057360 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17057360 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-15 04:13:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:21 proc_lsst.multi:170 [INFO] [multi] job_id 17057361 proc_lsst.multi INFO: [multi] job_id 17057361 2024-03-15 04:13:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057361 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057361 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17057361 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-15 04:13:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:21 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:22 proc_lsst.multi:170 [INFO] [multi] job_id 17057362 proc_lsst.multi INFO: [multi] job_id 17057362 2024-03-15 04:13:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 04:13:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057362 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057362 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17057362 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-15 04:13:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:22 proc_lsst.multi:170 [INFO] [multi] job_id 17057363 proc_lsst.multi INFO: [multi] job_id 17057363 2024-03-15 04:13:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 04:13:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057363 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057363 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17057363 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-15 04:13:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:22 proc_lsst.multi:170 [INFO] [multi] job_id 17057364 proc_lsst.multi INFO: [multi] job_id 17057364 2024-03-15 04:13:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 04:13:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057364 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057364 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17057364 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-15 04:13:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:22 proc_lsst.multi:170 [INFO] [multi] job_id 17057365 proc_lsst.multi INFO: [multi] job_id 17057365 2024-03-15 04:13:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 04:13:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057365 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057365 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17057365 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-15 04:13:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:22 proc_lsst.multi:170 [INFO] [multi] job_id 17057366 proc_lsst.multi INFO: [multi] job_id 17057366 2024-03-15 04:13:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 04:13:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057366 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057366 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17057366 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-15 04:13:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:22 proc_lsst.multi:170 [INFO] [multi] job_id 17057367 proc_lsst.multi INFO: [multi] job_id 17057367 2024-03-15 04:13:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 04:13:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057367 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057367 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17057367 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-15 04:13:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:22 proc_lsst.multi:170 [INFO] [multi] job_id 17057368 proc_lsst.multi INFO: [multi] job_id 17057368 2024-03-15 04:13:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 04:13:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057368 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057368 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17057368 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-15 04:13:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:22 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-15 04:13:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17057369 proc_lsst.multi INFO: [multi] job_id 17057369 2024-03-15 04:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 04:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057369 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057369 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17057369 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-15 04:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17057370 proc_lsst.multi INFO: [multi] job_id 17057370 2024-03-15 04:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 04:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057370 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057370 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17057370 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-15 04:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17057371 proc_lsst.multi INFO: [multi] job_id 17057371 2024-03-15 04:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057371 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057371 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17057371 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-15 04:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17057372 proc_lsst.multi INFO: [multi] job_id 17057372 2024-03-15 04:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-15 04:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057372 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057372 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17057372 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-15 04:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17057373 proc_lsst.multi INFO: [multi] job_id 17057373 2024-03-15 04:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-15 04:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057373 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057373 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17057373 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-15 04:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17057374 proc_lsst.multi INFO: [multi] job_id 17057374 2024-03-15 04:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-15 04:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057374 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057374 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17057374 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-15 04:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:23 proc_lsst.multi:170 [INFO] [multi] job_id 17057375 proc_lsst.multi INFO: [multi] job_id 17057375 2024-03-15 04:13:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-15 04:13:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057375 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057375 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17057375 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-15 04:13:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-15 04:13:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:24 proc_lsst.multi:170 [INFO] [multi] job_id 17057376 proc_lsst.multi INFO: [multi] job_id 17057376 2024-03-15 04:13:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-15 04:13:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057376 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057376 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17057376 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-15 04:13:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-15 04:13:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-15 04:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:24 proc_lsst.multi:170 [INFO] [multi] job_id 17057377 proc_lsst.multi INFO: [multi] job_id 17057377 2024-03-15 04:13:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-15 04:13:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057377 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057377 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17057377 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-15 04:13:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-15 04:13:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-15 04:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-15 04:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-15 04:13:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-15 04:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-15 04:13:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-15 04:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-15 04:13:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-15 04:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-15 04:13:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-15 04:13:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-15 04:13:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.65.9,198.48.92.26,n3009,127.0.0.1,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54751 --result_port=54091 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-15 04:13:24 proc_lsst.multi:170 [INFO] [multi] job_id 17057378 proc_lsst.multi INFO: [multi] job_id 17057378 2024-03-15 04:13:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-15 04:13:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-15 04:13:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17057378 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17057378 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17057378 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/92 running/pending blocks, and 57 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 1/92 running/pending blocks, and 57 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f85d4c578cac048d1e51e6c6c1b351d with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash ea6ea571279d27cbd6e4a91141b7528b parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/48e24161-0321-4824-b601-90b6e4909e3d_characterizeImage_994639_44.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/48e24161-0321-4824-b601-90b6e4909e3d_characterizeImage_994639_44.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/88cbbbd5-0c2d-4773-b30b-60a1fe1f6506_isr_994639_44.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/88cbbbd5-0c2d-4773-b30b-60a1fe1f6506_isr_994639_44.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7483c41303092f147a70dacdb465574 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 56470ec47a15008e2e94b07fe2526737 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/6b170b20-9261-4338-b6e7-d8f857536d87_characterizeImage_994766_31.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/6b170b20-9261-4338-b6e7-d8f857536d87_characterizeImage_994766_31.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/9ea46b6d-37c3-4976-9862-c6c8fc94e4f9_isr_994766_31.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/9ea46b6d-37c3-4976-9862-c6c8fc94e4f9_isr_994766_31.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9a580d5aba39bd2ec997178745c761f with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 0c480b101e15696aa976ad188dcde24e parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a2701b9f-845a-43f3-bafd-8d03975dedc0_characterizeImage_994765_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a2701b9f-845a-43f3-bafd-8d03975dedc0_characterizeImage_994765_2.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/14d95624-672e-4674-a471-037678a81501_isr_994765_2.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/14d95624-672e-4674-a471-037678a81501_isr_994765_2.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d835ccbcf71be74d2b9152b5b5d2aa01 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 31f19740899f46e07442fd0742cfe18b parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/282a1115-e0ce-4093-be93-fe8b48e5c268_characterizeImage_994638_33.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/282a1115-e0ce-4093-be93-fe8b48e5c268_characterizeImage_994638_33.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/29a6d054-151c-4de3-a08f-af8f88543990_isr_994638_33.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/29a6d054-151c-4de3-a08f-af8f88543990_isr_994638_33.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dba18e1e3d15b4208bbd32517c0a521 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash fbe614cecc42fa295a78a520eef82fe3 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0145a0e7-051d-4e00-a728-0fcfa04f55d6_characterizeImage_994640_20.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0145a0e7-051d-4e00-a728-0fcfa04f55d6_characterizeImage_994640_20.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3f2106e6-e635-4ed0-a015-ee1e25a6820f_isr_994640_20.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3f2106e6-e635-4ed0-a015-ee1e25a6820f_isr_994640_20.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7678c1e9edadc7d1f24b6e002c02c4c4 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 46f46c98c438ddde2adbd9e4e09bd124 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/244f3313-a596-4bc4-bce8-b4d74cf727b2_characterizeImage_994639_56.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/244f3313-a596-4bc4-bce8-b4d74cf727b2_characterizeImage_994639_56.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/62bb73cd-d163-4f91-a3d0-d08c9f1c943b_isr_994639_56.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/62bb73cd-d163-4f91-a3d0-d08c9f1c943b_isr_994639_56.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1331000d1de51e49e58bba4939863a6 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash d54aa956edc3fa2b7928b8668a8ebc72 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/89a01609-1000-493d-8db2-6c70d7a90970_characterizeImage_994767_10.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/89a01609-1000-493d-8db2-6c70d7a90970_characterizeImage_994767_10.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5e320610-41d1-43c3-8120-746447e7a29e_isr_994767_10.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5e320610-41d1-43c3-8120-746447e7a29e_isr_994767_10.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88955add29bf24c87ab9c07dbe796b4a with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 2f0deb82663594592cf371478a665341 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fafd5fd3-a9e0-4e67-ad82-fd08e256774a_characterizeImage_994767_40.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fafd5fd3-a9e0-4e67-ad82-fd08e256774a_characterizeImage_994767_40.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7033d20e-a0ba-4ea3-b19c-0389b2db2b68_isr_994767_40.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7033d20e-a0ba-4ea3-b19c-0389b2db2b68_isr_994767_40.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50c90bc88dbd1992b88b3634da7bc204 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash b13b15d540dd7186d87893aa8a5784e1 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c948c518-16d8-4586-8389-0790dcd867aa_characterizeImage_994765_45.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c948c518-16d8-4586-8389-0790dcd867aa_characterizeImage_994765_45.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/c122dce0-5bf9-4caa-9bed-448b0ae3a348_isr_994765_45.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/c122dce0-5bf9-4caa-9bed-448b0ae3a348_isr_994765_45.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e04f36a5fc0d74b724921d51f5a8bd56 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 967c176ebf751f39709b2346c01de317 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e3acbf1c-9646-440c-8c9b-daf90e306dc0_characterizeImage_994766_30.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e3acbf1c-9646-440c-8c9b-daf90e306dc0_characterizeImage_994766_30.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/1f94dca6-518a-4046-8230-19a73571dec6_isr_994766_30.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/1f94dca6-518a-4046-8230-19a73571dec6_isr_994766_30.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d71df77e91d3215ecc344c10f7532ac with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash b0462510048adf67e20161a214a3d430 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7640ae81-7729-4656-8ec6-bff6d86967ab_characterizeImage_994639_52.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7640ae81-7729-4656-8ec6-bff6d86967ab_characterizeImage_994639_52.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/60919b39-1912-4403-8355-67f22af220c2_isr_994639_52.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/60919b39-1912-4403-8355-67f22af220c2_isr_994639_52.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a76f6719b18d9feee395942a48d167dc with result from task 114 2024-03-15 04:13:34 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 036a162e83600cf0d5b730611a8a3abc parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5445d472-e3bb-4ee6-b3fb-2c77934ae913_characterizeImage_994639_47.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5445d472-e3bb-4ee6-b3fb-2c77934ae913_characterizeImage_994639_47.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/03ba2c47-b350-483c-9d39-7dc91f208712_isr_994639_47.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/03ba2c47-b350-483c-9d39-7dc91f208712_isr_994639_47.stderr 2024-03-15 04:13:34 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ff1c267b9def039350eb1a46ca73b2a with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash e2d985ee4b3d06f12595a2f0f82cdbf2 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e2c6b254-8c5b-4c5d-af81-80c6b6d71798_characterizeImage_994638_7.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e2c6b254-8c5b-4c5d-af81-80c6b6d71798_characterizeImage_994638_7.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fe6f5f0d-ad8e-4f81-8fba-05e6c311cc4d_isr_994638_7.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fe6f5f0d-ad8e-4f81-8fba-05e6c311cc4d_isr_994638_7.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d7fc75540d3994f38290b419684d5e4 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 8e8ce30f6bd93573fb5b63b60e3fe59e parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/75518ea5-d557-478d-bf11-1af26df7ac88_characterizeImage_994767_62.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/75518ea5-d557-478d-bf11-1af26df7ac88_characterizeImage_994767_62.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/eb66a2c3-4375-40eb-989f-994537f8013b_isr_994767_62.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/eb66a2c3-4375-40eb-989f-994537f8013b_isr_994767_62.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd4c493ce552fdcfa144c4ff4408445 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 794a669a240ef6d7aa6df1b92da30e43 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5593bc1a-e52a-49f5-b59d-f339d9bc4efe_characterizeImage_994766_10.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5593bc1a-e52a-49f5-b59d-f339d9bc4efe_characterizeImage_994766_10.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d589d1a6-d39d-4f61-8c95-2b6566b0b59d_isr_994766_10.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d589d1a6-d39d-4f61-8c95-2b6566b0b59d_isr_994766_10.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4dd4f6dde6614639ee084dc90a448d8 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 8307334fabc7afc493d9fe24b15b121e parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fddfe5b1-103c-4ad1-971f-6f795781e444_characterizeImage_994767_29.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fddfe5b1-103c-4ad1-971f-6f795781e444_characterizeImage_994767_29.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d3e43342-7ca3-481a-a980-eb4ddf9547f7_isr_994767_29.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d3e43342-7ca3-481a-a980-eb4ddf9547f7_isr_994767_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:34 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e49f6f07aa87dd8953c75221e14816a with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash ad9a2bf53f652e82e8f767927e8f75cc parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/1bb53827-df77-4889-96b0-c1738278d664_characterizeImage_994767_47.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/1bb53827-df77-4889-96b0-c1738278d664_characterizeImage_994767_47.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/b052eaab-3699-4ed7-946f-b819d31b7204_isr_994767_47.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/b052eaab-3699-4ed7-946f-b819d31b7204_isr_994767_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:35 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 942958f49bc53a3a3fecc0484fd14d26 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 5080fcd95e80dd4478b8b8e52d7b09c8 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b860c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/6aaee5dd-ea35-490d-9c5a-ba094687221d_characterizeImage_994638_28.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/6aaee5dd-ea35-490d-9c5a-ba094687221d_characterizeImage_994638_28.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6c593c56-4817-4ea2-ac03-e91f4a652b16_isr_994638_28.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6c593c56-4817-4ea2-ac03-e91f4a652b16_isr_994638_28.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4888a008b8fb1ea842710c5d22cd07a9 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash fe6a198b81ff6c561369a28b67192b0b parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b874c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b6f3a249-32ed-4c0c-8c2f-d30902698093_characterizeImage_994638_15.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b6f3a249-32ed-4c0c-8c2f-d30902698093_characterizeImage_994638_15.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/2eeddd72-9a51-495a-bc64-d9da823edd7f_isr_994638_15.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/2eeddd72-9a51-495a-bc64-d9da823edd7f_isr_994638_15.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 872c8bb6ba02c9d2e65b725890f1698f with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash d237256b88c21dbcf2d9b67187599c76 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5bf677e0-c7ec-4f95-9cc9-920385838dcd_characterizeImage_994640_30.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5bf677e0-c7ec-4f95-9cc9-920385838dcd_characterizeImage_994640_30.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d2803a60-65f4-4306-8753-d06a4fee97e0_isr_994640_30.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d2803a60-65f4-4306-8753-d06a4fee97e0_isr_994640_30.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 671827d6aae34ca22304ecec3fcdcb89 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 97d6abaa9412124f35d4119604627150 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b871a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8a19faaf-da84-4111-8019-b590d28fbcc8_characterizeImage_994640_46.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8a19faaf-da84-4111-8019-b590d28fbcc8_characterizeImage_994640_46.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e8ed626d-1a4f-438a-8806-ee3abc3cd3d3_isr_994640_46.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e8ed626d-1a4f-438a-8806-ee3abc3cd3d3_isr_994640_46.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61b4a360897f01cbaf85c7c7074517cb with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 2b942cd203baeb883cd4d960dffa7354 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b85940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/be8fc17c-d578-4566-b1c2-eaa94c2351f8_characterizeImage_994766_23.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/be8fc17c-d578-4566-b1c2-eaa94c2351f8_characterizeImage_994766_23.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4d5b323b-e795-40b4-bef9-b82fcecc2288_isr_994766_23.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4d5b323b-e795-40b4-bef9-b82fcecc2288_isr_994766_23.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb368e145d7a4e90c58949f50ddfe8ef with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash ac5da1fe87b7f6e0fb0dca8c43248205 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b87f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:35 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/71e6da64-dce3-40f2-81fe-cc41bab94d1e_characterizeImage_994767_27.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/71e6da64-dce3-40f2-81fe-cc41bab94d1e_characterizeImage_994767_27.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cc3eca27-4f1d-4ffc-a1a8-789e5fc8d061_isr_994767_27.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cc3eca27-4f1d-4ffc-a1a8-789e5fc8d061_isr_994767_27.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ea25aba757b0350b65b29f15603cfbc with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 94fa62c313de2afebd1ae0fcd59fad5a parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b87b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/30bc65c9-5e7e-4763-9f96-4ff7facbc830_characterizeImage_994767_14.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/30bc65c9-5e7e-4763-9f96-4ff7facbc830_characterizeImage_994767_14.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/70b24c54-5bbd-40b5-8a98-c9d36d348281_isr_994767_14.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/70b24c54-5bbd-40b5-8a98-c9d36d348281_isr_994767_14.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce98c3a51e5df43be1a1477265ce53fa with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 3ad7a711accb650d793cf7212e52a21a parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/be01c5a7-f01d-43fb-96ef-207f2bbfbacc_characterizeImage_994767_30.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/be01c5a7-f01d-43fb-96ef-207f2bbfbacc_characterizeImage_994767_30.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/b6f43b29-61a0-40b3-8202-3d06d092dab9_isr_994767_30.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/b6f43b29-61a0-40b3-8202-3d06d092dab9_isr_994767_30.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59aa64463faa0b59bc394dae28ab1837 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash dbe57d454688971f45671c0b3e87e7d1 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b867a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/fa9b9872-30c6-4d2f-aa00-4a3d63a87f6d_characterizeImage_994638_30.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/fa9b9872-30c6-4d2f-aa00-4a3d63a87f6d_characterizeImage_994638_30.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3a266abe-fd4e-4e34-a3ac-de3798aef6c0_isr_994638_30.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3a266abe-fd4e-4e34-a3ac-de3798aef6c0_isr_994638_30.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5607f504b8e32d8bec442aa8451f6e79 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 98233d33a3df4e18a704917294f84304 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8824782c-fce9-4e1f-b880-4b4091a1d20f_characterizeImage_994767_57.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8824782c-fce9-4e1f-b880-4b4091a1d20f_characterizeImage_994767_57.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/12ac353f-8a96-4e22-a42a-aaf376fb577b_isr_994767_57.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/12ac353f-8a96-4e22-a42a-aaf376fb577b_isr_994767_57.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07295527f2333430b05a4a865b555015 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 269b19cfddb2621e32d439501d283e6b parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/17526b9f-7075-490f-a988-09f0c5943b35_characterizeImage_994765_25.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/17526b9f-7075-490f-a988-09f0c5943b35_characterizeImage_994765_25.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/d353f3d9-a316-4b88-908c-addd16da83ef_isr_994765_25.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/d353f3d9-a316-4b88-908c-addd16da83ef_isr_994765_25.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8efd6df190046133507c0698d1709628 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 6fa2809f1491090837fe7cc4cd7abd65 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fa9d003e-e592-4347-9100-7dc62ed466a5_characterizeImage_994767_24.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fa9d003e-e592-4347-9100-7dc62ed466a5_characterizeImage_994767_24.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/898b6be7-ce81-4d31-ba4c-9f12bd1c8ed7_isr_994767_24.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/898b6be7-ce81-4d31-ba4c-9f12bd1c8ed7_isr_994767_24.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b811b2ed7567efd2039637c49ad4496 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 07eea748d3e65ee71cc48a4c513f87bf parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b85d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a9de49ab-5691-441c-ab9b-1fabf0c19137_characterizeImage_994638_50.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a9de49ab-5691-441c-ab9b-1fabf0c19137_characterizeImage_994638_50.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3a961bae-9655-4fb3-a4e2-907830049297_isr_994638_50.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3a961bae-9655-4fb3-a4e2-907830049297_isr_994638_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:36 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c02bb23b61d9f2099e0efa897901bedf with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 03721241ec80991856343a02cdfe0890 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0e0778a3-bc42-4093-988b-f6417f432a2f_characterizeImage_994640_3.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0e0778a3-bc42-4093-988b-f6417f432a2f_characterizeImage_994640_3.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ee22638a-57cc-497f-af06-15c143386431_isr_994640_3.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ee22638a-57cc-497f-af06-15c143386431_isr_994640_3.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 563ac828db84d6443dffe9d9e1e44750 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 184c68f8aad3ba9cd7a6cde831d50ddb parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/242fbb5c-2f75-49af-8341-5707745a7dde_characterizeImage_994640_41.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/242fbb5c-2f75-49af-8341-5707745a7dde_characterizeImage_994640_41.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/dfe48501-854d-463e-ae7c-1a1cc3536e53_isr_994640_41.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/dfe48501-854d-463e-ae7c-1a1cc3536e53_isr_994640_41.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb849dae51ddefef3d638896ed748583 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 4bd265e0f094b8d131af546869dab9dc parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b05291aa-d900-4b70-8140-a01d4e9f75ac_characterizeImage_994638_56.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b05291aa-d900-4b70-8140-a01d4e9f75ac_characterizeImage_994638_56.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ab1a351e-e0f8-40d6-8354-8b6ce61799b4_isr_994638_56.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ab1a351e-e0f8-40d6-8354-8b6ce61799b4_isr_994638_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:36 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70d548b3df830173aa4efca05d3993cf with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 527bb518657b74bae4f7c4624a7992c1 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7b7eda5e-521a-45ae-8876-eb78c477b726_characterizeImage_994639_28.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7b7eda5e-521a-45ae-8876-eb78c477b726_characterizeImage_994639_28.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/bcf1240d-cf86-4b33-8f7f-06834f764f7d_isr_994639_28.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/bcf1240d-cf86-4b33-8f7f-06834f764f7d_isr_994639_28.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d6b2722dbff22c3b04aee5eec93b95f with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 6cedda2b9df0a9e02d94359a832272ec parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/eebf294a-f6a1-4307-9de0-f95892bc4ae3_characterizeImage_994765_5.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/eebf294a-f6a1-4307-9de0-f95892bc4ae3_characterizeImage_994765_5.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fa401e05-9ccf-40ad-86c7-1bc6e4b17b9d_isr_994765_5.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fa401e05-9ccf-40ad-86c7-1bc6e4b17b9d_isr_994765_5.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2af3cf43db41e9cdbe44641d6339931d with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash efe234cab7b1b1bbbf6b8267b36f3bf8 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b877e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/27300754-fc7e-4fb7-bf51-3cebcc49aefc_characterizeImage_994639_12.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/27300754-fc7e-4fb7-bf51-3cebcc49aefc_characterizeImage_994639_12.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cf373f22-677e-497d-b1e7-7b23f445dc2b_isr_994639_12.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cf373f22-677e-497d-b1e7-7b23f445dc2b_isr_994639_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:37 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14c529155eb15b1ec9790587a2961a66 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 9381ea68baf78a0a02a038b1daf10432 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/52145c6d-1bea-40fd-8fcd-cbe82c3591d6_characterizeImage_994767_18.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/52145c6d-1bea-40fd-8fcd-cbe82c3591d6_characterizeImage_994767_18.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/039a5d61-4bcd-4d8e-b143-ace06fb92cd0_isr_994767_18.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/039a5d61-4bcd-4d8e-b143-ace06fb92cd0_isr_994767_18.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed98e00f106740672aac88a93e8c56eb with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash a34d534e7c5e45757b54ff791e66d483 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b85620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0688159a-5d62-4cd9-989e-fb6130a3a139_characterizeImage_994639_37.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0688159a-5d62-4cd9-989e-fb6130a3a139_characterizeImage_994639_37.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/f218bee6-7b15-497b-a39a-91adbe3afc0f_isr_994639_37.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/f218bee6-7b15-497b-a39a-91adbe3afc0f_isr_994639_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:37 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e53e04399b20e952bc854d8cede0158 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash cd5103fd4d51bcffdab14c7d331e2f42 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c1b80f70-805e-423a-a86e-eeb9f146c1e2_characterizeImage_994766_36.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c1b80f70-805e-423a-a86e-eeb9f146c1e2_characterizeImage_994766_36.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8ffc6d94-8a37-42b3-a710-c9a5956cdaca_isr_994766_36.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8ffc6d94-8a37-42b3-a710-c9a5956cdaca_isr_994766_36.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc484de66ada741bbfd094cc6153c315 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash aba9dff45f86fb802ab164cf739c91e9 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b87e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/be80aebe-6111-4735-81a6-999bb8e775b0_characterizeImage_994766_59.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/be80aebe-6111-4735-81a6-999bb8e775b0_characterizeImage_994766_59.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d4565c2c-df15-4dd2-a327-7ea7e37f81aa_isr_994766_59.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d4565c2c-df15-4dd2-a327-7ea7e37f81aa_isr_994766_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:38 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68d1850b81ec63c80726083fcba13575 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash d5274e808b91ea2a360534eb5829ddc6 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e91733ba-c9df-4351-9a1d-0e4fe56ad658_characterizeImage_994766_52.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e91733ba-c9df-4351-9a1d-0e4fe56ad658_characterizeImage_994766_52.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/f9a9dba0-865e-4c32-b963-4b190f2b7f78_isr_994766_52.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/f9a9dba0-865e-4c32-b963-4b190f2b7f78_isr_994766_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:38 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:38 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:38 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:39 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:39 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:39 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:40 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:40 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:40 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:41 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:41 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54760103a59003ed21a04a758af33a13 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash cd835953ab4e1f5e25d8a445c1d54de7 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a50a4ee7-2011-49ea-addc-7ad38ecc08ef_characterizeImage_994766_19.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a50a4ee7-2011-49ea-addc-7ad38ecc08ef_characterizeImage_994766_19.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ce3211b2-aa50-436d-ac4d-2a105dc1fda5_isr_994766_19.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ce3211b2-aa50-436d-ac4d-2a105dc1fda5_isr_994766_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:41 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03f49633298d1824648f5138c9c31028 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 1524161761b4e566e9e8af8462968960 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a6f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/97c7ceef-235c-4076-b0eb-b3b120a9c706_characterizeImage_994767_25.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/97c7ceef-235c-4076-b0eb-b3b120a9c706_characterizeImage_994767_25.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cd84988b-011f-4b8c-899c-360c90938ca6_isr_994767_25.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cd84988b-011f-4b8c-899c-360c90938ca6_isr_994767_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:42 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 276b6e2b7a8c01a47f3cd93e715aa81d with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash ecf58297fca2016453f0805511f025d5 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a6c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/53d7c85c-f3e3-4982-a721-f1581ccce5c9_characterizeImage_994638_17.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/53d7c85c-f3e3-4982-a721-f1581ccce5c9_characterizeImage_994638_17.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/c5722a1d-07b2-41b3-8bd8-8de2345a136c_isr_994638_17.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/c5722a1d-07b2-41b3-8bd8-8de2345a136c_isr_994638_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:42 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1625967d1316e3543c82d49f22a0f531 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 92e64f070709aaf4e2f71c3a6394333d parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c605c894-b887-4ee3-b08a-7f86c0569b8d_characterizeImage_994766_49.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c605c894-b887-4ee3-b08a-7f86c0569b8d_characterizeImage_994766_49.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/10329def-a9eb-493f-adfd-c4c6ecffcf05_isr_994766_49.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/10329def-a9eb-493f-adfd-c4c6ecffcf05_isr_994766_49.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb167a4192cbdef4792af931c19cec3d with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 81f0135719293e8d34f18ab82b9fc644 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a65c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7787956e-f9b7-423f-9fbc-163cdca04340_characterizeImage_994639_11.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7787956e-f9b7-423f-9fbc-163cdca04340_characterizeImage_994639_11.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/8a224f86-3d42-4ecd-a653-1e98370f82a2_isr_994639_11.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/8a224f86-3d42-4ecd-a653-1e98370f82a2_isr_994639_11.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb0b274a521e99c39ce4da41770d56b5 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 087da587767bc2422d9b47dfa562356a parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a62a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/4c4877dc-f21a-4357-93ae-0f8b857f40ad_characterizeImage_994639_22.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/4c4877dc-f21a-4357-93ae-0f8b857f40ad_characterizeImage_994639_22.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/0a1d8927-9458-4e81-bcb3-5927fc566d1b_isr_994639_22.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/0a1d8927-9458-4e81-bcb3-5927fc566d1b_isr_994639_22.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5be0810d79b7ac9b84c986c0676dc1c7 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 9d994abe82a78477079178fd46f3ed7f parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/cb63d270-880f-4f7f-9d3d-c25edae6250b_characterizeImage_994640_62.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/cb63d270-880f-4f7f-9d3d-c25edae6250b_characterizeImage_994640_62.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5dd1ce2b-16d5-46ab-945e-b5d4b7904bb5_isr_994640_62.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5dd1ce2b-16d5-46ab-945e-b5d4b7904bb5_isr_994640_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:42 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6a753e6ba11825e28f4e108877372d0 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 049e00e6dba93d66f53618929d980345 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/54f55163-541e-4af0-98a2-9aace25cf3cd_characterizeImage_994766_26.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/54f55163-541e-4af0-98a2-9aace25cf3cd_characterizeImage_994766_26.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/421f41c9-74c2-4355-8840-74ddc3dcb652_isr_994766_26.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/421f41c9-74c2-4355-8840-74ddc3dcb652_isr_994766_26.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63b07ed4272b2b878169ed9eaa243c75 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash aa0bc051e1e28e8cd94ce547aa9b23b5 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f7258449-3982-4313-96d2-3a60b3502a4c_characterizeImage_994765_52.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f7258449-3982-4313-96d2-3a60b3502a4c_characterizeImage_994765_52.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cc084427-0f25-4086-b7e8-e04c0bf37cb6_isr_994765_52.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cc084427-0f25-4086-b7e8-e04c0bf37cb6_isr_994765_52.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba6a16f9d92add0992ab10383cbaa0f2 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 1d6bd59fabfc3a126f2036a52812d4ab parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/95456d32-0010-4291-8d7a-7933366226ff_characterizeImage_994765_62.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/95456d32-0010-4291-8d7a-7933366226ff_characterizeImage_994765_62.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9228d7e2-6b2f-4de3-bc0d-52da472879e1_isr_994765_62.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9228d7e2-6b2f-4de3-bc0d-52da472879e1_isr_994765_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:43 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:43 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:44 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63a7c4c4effe4c4d4c51608faf866c8a with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash cd9bb27c69ef77ef5d2549148dccaf83 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a68e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/65495b36-7879-460a-a706-a4e34a4cc722_characterizeImage_994638_59.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/65495b36-7879-460a-a706-a4e34a4cc722_characterizeImage_994638_59.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ca208ad2-409f-47cb-962d-35e3073feaa8_isr_994638_59.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ca208ad2-409f-47cb-962d-35e3073feaa8_isr_994638_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:44 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:45 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7c75b1abd51f704b247b901d4e4999b with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 1601d8a69ab62bfb749f248edebfb298 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/74c76584-8472-444e-a19c-ad80322e505e_characterizeImage_994639_21.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/74c76584-8472-444e-a19c-ad80322e505e_characterizeImage_994639_21.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/a2a18200-0342-450d-810a-d1a3d2a13028_isr_994639_21.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/a2a18200-0342-450d-810a-d1a3d2a13028_isr_994639_21.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c72c726455a749342fd948930190bf2 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash c7ea77dd2751cd142c193660d74b4eac parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f9f72975-c9f5-47b2-9362-688945ddb623_characterizeImage_994766_21.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f9f72975-c9f5-47b2-9362-688945ddb623_characterizeImage_994766_21.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/a24df6a5-be4a-47f7-812f-99dacd1be0d5_isr_994766_21.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/a24df6a5-be4a-47f7-812f-99dacd1be0d5_isr_994766_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:45 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:45 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:46 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:46 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:47 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 251cd960ce63604bfe5dcb25d3c7ed4e with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 2cdaf75c7f7cff6c60ab22ee2738f5d0 parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/919854bf-d625-46cd-93f9-7412933f08be_characterizeImage_994639_31.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/919854bf-d625-46cd-93f9-7412933f08be_characterizeImage_994639_31.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/27b97d78-c311-46d4-a9b2-19d3bb4abadd_isr_994639_31.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/27b97d78-c311-46d4-a9b2-19d3bb4abadd_isr_994639_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:47 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10a2ac4c1f0133f8c517120c92cadfe7 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 112baadff15761f3482fd62702ac7b7d parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c2bba4cb-c1b8-46fa-a23b-c4cedbe44421_characterizeImage_994767_51.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c2bba4cb-c1b8-46fa-a23b-c4cedbe44421_characterizeImage_994767_51.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/a9318e72-ac87-4f39-abae-304dfb671f5d_isr_994767_51.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/a9318e72-ac87-4f39-abae-304dfb671f5d_isr_994767_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:47 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d04ae40fe5f2c75670aa7318a0bc453 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash c73419e5d6453394c9c50e15dc74e29e parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/2bdaaedc-4645-4a31-a8c5-5ff53bcb7b8b_characterizeImage_994639_53.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/2bdaaedc-4645-4a31-a8c5-5ff53bcb7b8b_characterizeImage_994639_53.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d8f896c8-6d57-4c96-a0d3-22c9616794bc_isr_994639_53.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d8f896c8-6d57-4c96-a0d3-22c9616794bc_isr_994639_53.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dd3293bb7022ab1cab8cf8b75591f87 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 51af2f7821ec11877b3e8aa3696118cd parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a16849d7-356f-4dfa-87f9-3a31f184a365_characterizeImage_994765_27.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a16849d7-356f-4dfa-87f9-3a31f184a365_characterizeImage_994765_27.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87aa2a24-fadc-4f98-9aca-3fc287ac047f_isr_994765_27.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87aa2a24-fadc-4f98-9aca-3fc287ac047f_isr_994765_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:48 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:48 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a83760de46961b18ad9f48340e4026c with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 0f709a79e6293d34f298625fcf8169b1 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/90809a26-b2bc-46e1-85b9-d8881f1f7c89_characterizeImage_994766_43.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/90809a26-b2bc-46e1-85b9-d8881f1f7c89_characterizeImage_994766_43.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/01471a38-6093-4ace-b1d3-9c127ccdab21_isr_994766_43.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/01471a38-6093-4ace-b1d3-9c127ccdab21_isr_994766_43.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e39c08435370c7785b9db155cecd355c with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 09e535754fbb0ea02f62f04a4a2d317f parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4817a43b-a354-4e8e-a07b-3d7f9c413ab3_characterizeImage_994767_16.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4817a43b-a354-4e8e-a07b-3d7f9c413ab3_characterizeImage_994767_16.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/6f135eb2-0613-41eb-acfb-6860ac3e6a20_isr_994767_16.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/6f135eb2-0613-41eb-acfb-6860ac3e6a20_isr_994767_16.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d864f87a67029dd43d561a5b8f6dd48c with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash f4864e005fe8f1007b965c544a4a0c08 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/9c545647-93df-4a6d-949d-6755eefee481_characterizeImage_994767_53.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/9c545647-93df-4a6d-949d-6755eefee481_characterizeImage_994767_53.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/fec2937b-a826-441d-a583-8ef5389a6878_isr_994767_53.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/fec2937b-a826-441d-a583-8ef5389a6878_isr_994767_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:48 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c89205f02f219a9f203ea7021b7d3882 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash f84430394ccfdbc7a3d7ebba6f3942d0 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/fb748477-817e-4f2e-a9f0-e2a45d6b6b96_characterizeImage_994640_39.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/fb748477-817e-4f2e-a9f0-e2a45d6b6b96_characterizeImage_994640_39.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ef4b5386-dc56-4461-aaf1-dde7b509de48_isr_994640_39.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ef4b5386-dc56-4461-aaf1-dde7b509de48_isr_994640_39.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26f64692ba069c8e7423638d8c574af3 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 229d9764ce103cdcf0bae98edbbc19df parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/a075fb17-8daf-4767-8818-085f36bca678_characterizeImage_994767_61.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/a075fb17-8daf-4767-8818-085f36bca678_characterizeImage_994767_61.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/e5045079-dd41-481b-b5f3-143916e2f8b6_isr_994767_61.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/e5045079-dd41-481b-b5f3-143916e2f8b6_isr_994767_61.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82efc9f334f5e7a7bae5704fe99fe86f with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 442eec3f5d280a648d2d7e3d2183c9cd parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a77a7c89-7273-4ea6-993d-f5f1f9192ca9_characterizeImage_994638_27.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a77a7c89-7273-4ea6-993d-f5f1f9192ca9_characterizeImage_994638_27.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3b826cce-41e7-48b3-9436-12cd1348043e_isr_994638_27.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3b826cce-41e7-48b3-9436-12cd1348043e_isr_994638_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:48 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c98319d5d0dad355623c10a48d21a35 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 0af3730abc996e453b81c2c8487dcca1 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c7975d08-5394-491d-99aa-da315035f3da_characterizeImage_994766_16.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c7975d08-5394-491d-99aa-da315035f3da_characterizeImage_994766_16.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/30904626-c451-4139-93da-adb9340bd04e_isr_994766_16.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/30904626-c451-4139-93da-adb9340bd04e_isr_994766_16.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 082d4115f9c04b4ada1b0fc173a91f13 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 1de3f59e1196a0683aa802d67b774522 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a3e0f48f-8a9a-43c3-804d-ffdf02488b81_characterizeImage_994638_48.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a3e0f48f-8a9a-43c3-804d-ffdf02488b81_characterizeImage_994638_48.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6f592b24-bcfe-4a8d-a754-4e936d8f14e1_isr_994638_48.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6f592b24-bcfe-4a8d-a754-4e936d8f14e1_isr_994638_48.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0e488d3d1723314f42a83f290a096e8 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash e25a7336f1c8494a8914024c2739e4e4 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/df761ce4-748c-484f-8194-987e91cb2a2d_characterizeImage_994766_7.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/df761ce4-748c-484f-8194-987e91cb2a2d_characterizeImage_994766_7.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/cf18ae22-edf4-41b7-be29-1a0e49ff6b98_isr_994766_7.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/cf18ae22-edf4-41b7-be29-1a0e49ff6b98_isr_994766_7.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 954bb4661153a2741660d50801051e12 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 0fd28d6e1b5adf78c9cfc702f4d1f7d0 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2421ba9d-8a04-4d6d-ba4a-b249fd90fd81_characterizeImage_994766_60.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2421ba9d-8a04-4d6d-ba4a-b249fd90fd81_characterizeImage_994766_60.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/22dde8c5-bfff-491c-925c-489650f7e8dd_isr_994766_60.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/22dde8c5-bfff-491c-925c-489650f7e8dd_isr_994766_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:49 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15f3894ad24e1ab8ca607d4474b06be6 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash b9d18afef13889413e6934b547d684d2 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e53a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7c399678-9753-4a78-b508-09d4f5ac42c5_characterizeImage_994639_24.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7c399678-9753-4a78-b508-09d4f5ac42c5_characterizeImage_994639_24.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/611c5128-6680-4e07-b582-8bcecb57a9a7_isr_994639_24.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/611c5128-6680-4e07-b582-8bcecb57a9a7_isr_994639_24.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98e508cd5af052e17aaee6cfa9e12798 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 6fead8dd5762df4effdb885765115bc5 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/53947b9a-cfff-438b-aff7-a2e7d53bd0c6_characterizeImage_994767_4.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/53947b9a-cfff-438b-aff7-a2e7d53bd0c6_characterizeImage_994767_4.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c8442a62-d3b4-402b-b700-e87b3e7b280f_isr_994767_4.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c8442a62-d3b4-402b-b700-e87b3e7b280f_isr_994767_4.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f666598ab2383cebdd767099d8817f97 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash acec8d2945e9d2a90784e20be962b3a6 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/8f6aa5d7-1845-4ae0-81bd-a16d1f435a34_characterizeImage_994638_4.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/8f6aa5d7-1845-4ae0-81bd-a16d1f435a34_characterizeImage_994638_4.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ec15d7d1-7e7d-4fe0-907e-33e1d3955a50_isr_994638_4.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ec15d7d1-7e7d-4fe0-907e-33e1d3955a50_isr_994638_4.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 421773a13ccf9043259068a262417b7e with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 0825a24751e227d2ea4106e7a4b176d5 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/b6f85c45-7d01-47ec-96b2-a7bb1af2a1fa_characterizeImage_994766_18.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/b6f85c45-7d01-47ec-96b2-a7bb1af2a1fa_characterizeImage_994766_18.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/19cc2a90-1314-4516-9945-764708f2d164_isr_994766_18.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/19cc2a90-1314-4516-9945-764708f2d164_isr_994766_18.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2042da792874e1401ec9771ef121daf with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 67b1e8d0395901780bc9fc33afaf5a86 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ac551ee4-127c-4dc0-a8bd-b6116e8f8aa5_characterizeImage_994639_18.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ac551ee4-127c-4dc0-a8bd-b6116e8f8aa5_characterizeImage_994639_18.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/485a8ee6-30de-489f-bbbd-622a67e5e4d0_isr_994639_18.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/485a8ee6-30de-489f-bbbd-622a67e5e4d0_isr_994639_18.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bcf51c4b5dc078aaa353e049c6ab362 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash ba4350bc5c89876c7b157f2089ad5227 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/04fe4f2b-a462-4d9a-a353-0d3e031dc134_characterizeImage_994766_20.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/04fe4f2b-a462-4d9a-a353-0d3e031dc134_characterizeImage_994766_20.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/82b1bca8-07fb-4ad7-a137-9118e09d2b3e_isr_994766_20.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/82b1bca8-07fb-4ad7-a137-9118e09d2b3e_isr_994766_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:49 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aca97f5d1b55968e0a3f2df54ba34368 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash ad2c7d03cc304b2f36f7aa546f7472ce parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/1c67a1d6-f76d-49e4-bd57-034762ec7bc2_characterizeImage_994638_36.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/1c67a1d6-f76d-49e4-bd57-034762ec7bc2_characterizeImage_994638_36.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7e41b692-1b2c-4892-8687-71d18a3d9517_isr_994638_36.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7e41b692-1b2c-4892-8687-71d18a3d9517_isr_994638_36.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8a14e6011fa73cc2ced9a03313a4d68 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 8ed47087be1e50b00ea2d8060c710066 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5d5807d7-89ea-47b9-b48e-55ae38db2fe6_characterizeImage_994640_40.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5d5807d7-89ea-47b9-b48e-55ae38db2fe6_characterizeImage_994640_40.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5b719e66-c880-4d76-93bf-f64591371e6f_isr_994640_40.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5b719e66-c880-4d76-93bf-f64591371e6f_isr_994640_40.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64e7215cf967dee4c4be90790f6cd623 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 4d5fddca7ca0de905eb576f7a705717c parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e59e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/77861902-727d-43fa-b1ea-defa7adc7cf5_characterizeImage_994640_4.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/77861902-727d-43fa-b1ea-defa7adc7cf5_characterizeImage_994640_4.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c8e0d60e-a6f1-47c6-bb0e-5b896d812397_isr_994640_4.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c8e0d60e-a6f1-47c6-bb0e-5b896d812397_isr_994640_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:49 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cced51435d9ed47f558b1bfb44c363b with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 2480c60ef97d9b929228d1936886291b parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e56c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/88039974-223a-4da6-8ede-b0388fea1d0b_characterizeImage_994767_34.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/88039974-223a-4da6-8ede-b0388fea1d0b_characterizeImage_994767_34.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c1992084-a1c0-4db7-ad1c-7584a392c410_isr_994767_34.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c1992084-a1c0-4db7-ad1c-7584a392c410_isr_994767_34.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 321f5649d12fc69597c7ec732d65bab9 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash a80656f99130d4a496d0033330e61c38 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6600900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a8359622-4990-4bae-bf87-5d9f1b973951_characterizeImage_994766_11.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a8359622-4990-4bae-bf87-5d9f1b973951_characterizeImage_994766_11.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/e0d33581-2b45-4073-9e31-3062fb5a4676_isr_994766_11.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/e0d33581-2b45-4073-9e31-3062fb5a4676_isr_994766_11.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46d52481725a5da985d33d5f47b6de2e with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash 1fd0e69db55a12c3082a374769c64c7d parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66002c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/347962fb-3d86-491d-b465-43139a4289dc_characterizeImage_994639_16.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/347962fb-3d86-491d-b465-43139a4289dc_characterizeImage_994639_16.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/a2dcad45-97d9-48b3-928e-b8172925d47f_isr_994639_16.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/a2dcad45-97d9-48b3-928e-b8172925d47f_isr_994639_16.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5622bba8e3198259ad44fa9324262529 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash c5f22dada2479fa733d405ed015a230e parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/03996248-ab55-46b5-85c1-694a689a9830_characterizeImage_994765_50.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/03996248-ab55-46b5-85c1-694a689a9830_characterizeImage_994765_50.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/6b38b6dc-2371-4e1f-8cb9-ddd379235126_isr_994765_50.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/6b38b6dc-2371-4e1f-8cb9-ddd379235126_isr_994765_50.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a70b795e575ae099f38b882bbc8a3a71 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 8d4988c2172ec7cf6fc1b3ba86887df6 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/80ef293c-15f0-4a59-be36-8fa3f2ea831e_characterizeImage_994639_6.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/80ef293c-15f0-4a59-be36-8fa3f2ea831e_characterizeImage_994639_6.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d54c8b33-3abf-4889-a60d-3dbe91b43d6a_isr_994639_6.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d54c8b33-3abf-4889-a60d-3dbe91b43d6a_isr_994639_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:50 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2f3884b8304a4441094f2432fbed4af with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 607ec638c3a9ecab435ac49ad54657c4 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c55f3d39-91aa-4ee7-800e-e0e9200c3b82_characterizeImage_994639_14.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c55f3d39-91aa-4ee7-800e-e0e9200c3b82_characterizeImage_994639_14.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/63217d78-3fbf-4f35-9993-ca9d7e75278f_isr_994639_14.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/63217d78-3fbf-4f35-9993-ca9d7e75278f_isr_994639_14.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27f844831e808d7737399a82338f0124 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 83bb2494bdb9fb83b5a4aa61e672f6db parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c7310e47-392a-4f8c-b329-c2a120a67a33_characterizeImage_994640_43.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c7310e47-392a-4f8c-b329-c2a120a67a33_characterizeImage_994640_43.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e8430d30-5011-4eb6-ba79-da623e553428_isr_994640_43.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e8430d30-5011-4eb6-ba79-da623e553428_isr_994640_43.stderr parsl.dataflow.dflow INFO: Task 1095 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbec4e44ad131196951cc41c85057e5a with result from task 1095 parsl.dataflow.dflow DEBUG: Task 1095 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1096 has memoization hash 6719d0e190ff857546d5d72ad0c30b5f parsl.dataflow.memoization INFO: Task 1096 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1096 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d58975b9-e486-4999-971d-37ed98983e80_characterizeImage_994765_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d58975b9-e486-4999-971d-37ed98983e80_characterizeImage_994765_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/93328619-5ad8-40ea-9b01-709f1c33705d_isr_994765_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1095 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/93328619-5ad8-40ea-9b01-709f1c33705d_isr_994765_23.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 935b43ac28e2b149b0df4d346d44c81c with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 286e16cd60af0a92da8aa33911472cbd parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/91fc2fde-3241-400e-a1b4-65e4d8e75f4a_characterizeImage_994640_53.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/91fc2fde-3241-400e-a1b4-65e4d8e75f4a_characterizeImage_994640_53.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e03e3aa5-6494-4b8c-84a8-6fa5194c2da5_isr_994640_53.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e03e3aa5-6494-4b8c-84a8-6fa5194c2da5_isr_994640_53.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 037dee1af6e674d65b2181f3a9f5e4c4 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 06749810c92d60c62ada5554d6c097dc parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/7387aa2c-5f3e-452c-8877-b8c5876c4c42_characterizeImage_994766_47.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/7387aa2c-5f3e-452c-8877-b8c5876c4c42_characterizeImage_994766_47.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/aa277727-0ec4-4ec7-a80d-6cab5dee82e7_isr_994766_47.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/aa277727-0ec4-4ec7-a80d-6cab5dee82e7_isr_994766_47.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b28e7ec5c241d2cabb617e4ab50dba65 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash dd8cef8f385d3946255d95121e42d8e5 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/bbf532fd-b578-45f7-acf6-ad4fc7bd1378_characterizeImage_994638_60.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/bbf532fd-b578-45f7-acf6-ad4fc7bd1378_characterizeImage_994638_60.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7be9a731-982a-45ea-8b2b-ca5bdff61252_isr_994638_60.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7be9a731-982a-45ea-8b2b-ca5bdff61252_isr_994638_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:50 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:50 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67462de9bc3762f86c72e449254fa106 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash ac38784094ca88ed93a64304704c35f9 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6600f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/4beb5e42-ae90-446a-a7f5-4bc1549682d3_characterizeImage_994765_19.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/4beb5e42-ae90-446a-a7f5-4bc1549682d3_characterizeImage_994765_19.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9590beb9-2129-4a45-94f1-c236543ea00d_isr_994765_19.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9590beb9-2129-4a45-94f1-c236543ea00d_isr_994765_19.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f7250678283023a015b212d28eebbae with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 045e7b0d20b3ce1b23cf984821827abe parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ca0b2d9a-6ba8-4b7d-bd66-3daa2e829e82_characterizeImage_994640_52.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ca0b2d9a-6ba8-4b7d-bd66-3daa2e829e82_characterizeImage_994640_52.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/36a22edd-f226-4ff9-a3db-30be9e113260_isr_994640_52.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/36a22edd-f226-4ff9-a3db-30be9e113260_isr_994640_52.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb3f036e666bfe08c04b92c7d9a55f28 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 7ff210479073c100b3a019c1eadbde58 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/afa2e822-1850-4ee3-bd38-b5b81e921ca2_characterizeImage_994638_29.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/afa2e822-1850-4ee3-bd38-b5b81e921ca2_characterizeImage_994638_29.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fdde7ab5-f068-41b0-b0b1-0e211486bc70_isr_994638_29.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fdde7ab5-f068-41b0-b0b1-0e211486bc70_isr_994638_29.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7806695c4970bfc5db91dcac91b1ced with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 4ac97836031dcbc81c9e0da0971da3ea parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b6dddd21-96f1-4ed0-ac77-0024241991df_characterizeImage_994639_3.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b6dddd21-96f1-4ed0-ac77-0024241991df_characterizeImage_994639_3.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/5abc8802-22a0-4bb4-8f78-cf55c21c0c53_isr_994639_3.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/5abc8802-22a0-4bb4-8f78-cf55c21c0c53_isr_994639_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:50 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c6573af43972f8da8221153eaa91124 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 9795a8fe07cc6706d4e5bca519a72fee parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/43de96a2-e1be-44d4-b15b-75a79e13a43f_characterizeImage_994767_8.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/43de96a2-e1be-44d4-b15b-75a79e13a43f_characterizeImage_994767_8.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/958785ff-0498-43ee-8914-22dd15dc5af3_isr_994767_8.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/958785ff-0498-43ee-8914-22dd15dc5af3_isr_994767_8.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42edd5e9a793b594b20e9e5644d0f36e with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash b7d9b3df96f96b08ff99e0298925e824 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2de9f952-62ed-48c2-a3f4-555ff942ea9f_characterizeImage_994766_53.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2de9f952-62ed-48c2-a3f4-555ff942ea9f_characterizeImage_994766_53.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/623dc85f-6956-45e8-bb39-f262d0e202df_isr_994766_53.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/623dc85f-6956-45e8-bb39-f262d0e202df_isr_994766_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:51 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c980747289d13250952f2f6f26d33e22 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash e249de90cd6e4aecce6c65b260821799 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/7595d8e4-146b-4179-ba51-6b61377157bd_characterizeImage_994765_60.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/7595d8e4-146b-4179-ba51-6b61377157bd_characterizeImage_994765_60.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/27915628-10c8-4b09-bbef-7ae3c11060fa_isr_994765_60.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/27915628-10c8-4b09-bbef-7ae3c11060fa_isr_994765_60.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 545b7ac2ac571e110d796a8eebc41d20 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash ca92604bed00eb643369490c6ecb372f parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/27c87c3e-cf08-43c5-92a9-d52a19d5f28c_characterizeImage_994767_11.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/27c87c3e-cf08-43c5-92a9-d52a19d5f28c_characterizeImage_994767_11.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8f3de220-1ce6-4bb4-82c5-846b94728bff_isr_994767_11.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8f3de220-1ce6-4bb4-82c5-846b94728bff_isr_994767_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:51 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f1ebfd310bded22b1805e992ea088c9 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 377cdb7b86bbfbedde7f0fe3ecf2a6f3 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/88cb4f59-b422-4109-93f2-09046347d6b1_characterizeImage_994639_15.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/88cb4f59-b422-4109-93f2-09046347d6b1_characterizeImage_994639_15.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6db45845-4034-4b34-b77a-2b93a0a5c0ce_isr_994639_15.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6db45845-4034-4b34-b77a-2b93a0a5c0ce_isr_994639_15.stderr parsl.dataflow.dflow INFO: Task 1068 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e68d0443a0127f8b3701028eaaa0b0d3 with result from task 1068 parsl.dataflow.dflow DEBUG: Task 1068 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1069 has memoization hash 44f8cac5fca181cc3e0e53e474fa1da9 parsl.dataflow.memoization INFO: Task 1069 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62eeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1069 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/acfbc825-408f-4902-a04d-a3ab352d8b34_characterizeImage_994767_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/acfbc825-408f-4902-a04d-a3ab352d8b34_characterizeImage_994767_44.stderr parsl.dataflow.dflow INFO: Standard output for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/f1e8aecd-aad3-4476-b17c-a2d43bcc93c6_isr_994767_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1068 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/f1e8aecd-aad3-4476-b17c-a2d43bcc93c6_isr_994767_44.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06df3f85ca39c8b318c7c5040e3e3ab3 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash bb46b5db4a93350e34a5dbb8f6e00314 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/297069e3-3409-4025-893e-94066b537c25_characterizeImage_994640_58.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/297069e3-3409-4025-893e-94066b537c25_characterizeImage_994640_58.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e4cc5dc6-fb2f-4b95-8b54-11be2860b4f1_isr_994640_58.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e4cc5dc6-fb2f-4b95-8b54-11be2860b4f1_isr_994640_58.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 861f762d7efbeb02b0f88f8f215fa359 with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash 8e646f26a21cd807db1e7c786fd62ec9 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64872e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/f38c87de-bcf2-4fb4-a14e-622b553eea5e_characterizeImage_994640_32.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/f38c87de-bcf2-4fb4-a14e-622b553eea5e_characterizeImage_994640_32.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/04d1f5e0-9162-4a19-a96a-e568ccdc8b4c_isr_994640_32.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/04d1f5e0-9162-4a19-a96a-e568ccdc8b4c_isr_994640_32.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 734f9ae000d771806a09f125eaa29e30 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash f173a4bd1a6bf85c1b8a7f3868ebdff0 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/862176bb-9c7a-4f60-8377-54be2678340e_characterizeImage_994765_57.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/862176bb-9c7a-4f60-8377-54be2678340e_characterizeImage_994765_57.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0ce3815b-4af3-40cc-a7f5-db01efa43b25_isr_994765_57.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0ce3815b-4af3-40cc-a7f5-db01efa43b25_isr_994765_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:51 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad3e2fc6d90126675a37fb208f590bdb with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 5a3df981f82fcf0b575cee180c9dd409 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/304d2c1e-124c-4f83-8008-4f10093e9a3f_characterizeImage_994767_37.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/304d2c1e-124c-4f83-8008-4f10093e9a3f_characterizeImage_994767_37.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/390fc59b-7d5a-478d-9f8f-70e2deea4da8_isr_994767_37.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/390fc59b-7d5a-478d-9f8f-70e2deea4da8_isr_994767_37.stderr parsl.dataflow.dflow INFO: Task 1101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5817bc9ee66c9fca161c5ee81809c5b3 with result from task 1101 parsl.dataflow.dflow DEBUG: Task 1101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1102 has memoization hash b526231cf749509378a699a6cfa48f86 parsl.dataflow.memoization INFO: Task 1102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1102 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/cedd3a32-3626-4d4e-9a06-260e5358258a_characterizeImage_994638_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/cedd3a32-3626-4d4e-9a06-260e5358258a_characterizeImage_994638_1.stderr parsl.dataflow.dflow INFO: Standard output for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7f0d204f-4fbc-4aa8-838d-2c30d6743a73_isr_994638_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7f0d204f-4fbc-4aa8-838d-2c30d6743a73_isr_994638_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:51 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcc17175836bbe44966760036485fca6 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 4855b68326bb96a88e3597f3783d672a parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/45f6c1a2-4c41-401a-bcfe-df7211ec983d_characterizeImage_994767_31.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/45f6c1a2-4c41-401a-bcfe-df7211ec983d_characterizeImage_994767_31.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/2a135c37-2726-49ae-ada0-c93ae1eafd09_isr_994767_31.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/2a135c37-2726-49ae-ada0-c93ae1eafd09_isr_994767_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:51 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b64fc47c45aed39b5da6e0ff4e0eeb2b with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash e7c51c0c68fac9647e260f9b964fb2e9 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66005e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fb45ea79-9132-4051-ba2c-4d049b154b69_characterizeImage_994767_55.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fb45ea79-9132-4051-ba2c-4d049b154b69_characterizeImage_994767_55.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7df265da-10ed-4f90-905a-8f9c19b60da5_isr_994767_55.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7df265da-10ed-4f90-905a-8f9c19b60da5_isr_994767_55.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9c969e9bd69cfb45722d009c68833bd with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash 91d7f370edf2ce24c6183f298c3c45a3 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6600c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c700767c-5a5e-44f1-aa42-bf9e054cdf87_characterizeImage_994639_39.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c700767c-5a5e-44f1-aa42-bf9e054cdf87_characterizeImage_994639_39.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/01373479-0032-42cc-9542-0152d326ebf2_isr_994639_39.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/01373479-0032-42cc-9542-0152d326ebf2_isr_994639_39.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30301fb3a26261fd1a63d0f0848b48e2 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash e6c794ecc270a3ef423c9cea0a74284f parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:51 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3db4237a-f4fe-4308-935f-3114402d39da_characterizeImage_994639_9.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3db4237a-f4fe-4308-935f-3114402d39da_characterizeImage_994639_9.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/89b8e7c6-7bb6-4842-ab54-cbbdbd293042_isr_994639_9.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/89b8e7c6-7bb6-4842-ab54-cbbdbd293042_isr_994639_9.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d07d3da78f7cb5761353a4d9dcdf016 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 4b114e0483e75a6bb5abefb6df81243f parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/278fa85a-6a39-46c7-8d0d-ab81047db64a_characterizeImage_994765_34.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/278fa85a-6a39-46c7-8d0d-ab81047db64a_characterizeImage_994765_34.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a00c1c83-c640-4eff-9bd7-31d6fb738a6d_isr_994765_34.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a00c1c83-c640-4eff-9bd7-31d6fb738a6d_isr_994765_34.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d535630f7194437965aa580c1bfd8ddf with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 06073525f5a05a92fc4b9296bf5ca14c parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e5080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8b6a0fac-30da-4ca7-bb00-298409a0de73_characterizeImage_994639_59.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8b6a0fac-30da-4ca7-bb00-298409a0de73_characterizeImage_994639_59.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/0ec2e5b0-70f3-4824-956f-e711c18240d3_isr_994639_59.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/0ec2e5b0-70f3-4824-956f-e711c18240d3_isr_994639_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:52 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:52 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:52 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:53 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be4e42dbb9cd1b5e9e3e91c690136da9 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash bcf146fba7299e82262728da368ed89a parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66018a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0faf4069-2af1-4f46-9843-7243775fed0a_characterizeImage_994767_60.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0faf4069-2af1-4f46-9843-7243775fed0a_characterizeImage_994767_60.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ebdb1deb-eb35-42d7-8d57-03f16d42a393_isr_994767_60.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ebdb1deb-eb35-42d7-8d57-03f16d42a393_isr_994767_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:53 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6160c773c02b476e5253c35007decb62 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash a0172e54e45dc7df3cd3809834a67eb2 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/277a4ab8-af3e-47c1-80c6-6c3a0c5272c9_characterizeImage_994639_51.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/277a4ab8-af3e-47c1-80c6-6c3a0c5272c9_characterizeImage_994639_51.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d7b53daf-8d68-46fe-b5d6-e8e0dfcb72fe_isr_994639_51.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d7b53daf-8d68-46fe-b5d6-e8e0dfcb72fe_isr_994639_51.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d602b8311d15d44b99842b5bc914c50 with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 997ea847b23e772cdcdb98609a0b1ac3 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e87f76d3-1358-4099-9a05-d3f84cea725e_characterizeImage_994640_15.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e87f76d3-1358-4099-9a05-d3f84cea725e_characterizeImage_994640_15.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ce89154c-ccf6-4b3d-8611-7069b8b2b1f3_isr_994640_15.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/ce89154c-ccf6-4b3d-8611-7069b8b2b1f3_isr_994640_15.stderr parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5da7725739e52de5730a6c2707baeccd with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 59c643ab7fb52346a0c7ddd4489a1f8d parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6506520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/0220d844-dc44-4c17-946f-9432c00bbb02_characterizeImage_994765_51.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/0220d844-dc44-4c17-946f-9432c00bbb02_characterizeImage_994765_51.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b0d45f28-e641-48fb-83a2-29cd1729c4e3_isr_994765_51.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b0d45f28-e641-48fb-83a2-29cd1729c4e3_isr_994765_51.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8893b4779b34d2e004d83317d1afebc with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 09d8f845c136bdb1efa1c4338d94dec0 parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/147d1f36-cdeb-4f1b-abdb-8730758decdf_characterizeImage_994766_32.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/147d1f36-cdeb-4f1b-abdb-8730758decdf_characterizeImage_994766_32.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4cfe4753-1e27-49ab-8c24-413af7ebff68_isr_994766_32.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4cfe4753-1e27-49ab-8c24-413af7ebff68_isr_994766_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:54 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f82722fecb814ca72c0396486f904ead with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash d97c602b35d5ba954322576f31c08c62 parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65062a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b7b9dbb5-ffe2-4bb5-b208-63ffae20188e_characterizeImage_994765_18.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b7b9dbb5-ffe2-4bb5-b208-63ffae20188e_characterizeImage_994765_18.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/3ca75a10-8391-43c7-bfb9-a9faa14b3e8e_isr_994765_18.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/3ca75a10-8391-43c7-bfb9-a9faa14b3e8e_isr_994765_18.stderr parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11810f6670f7ec458087469545fe8af5 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash b086ad4497a002f05226e8c245c9b757 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8de89558-ac38-49bb-b103-3219f4c9a3c2_characterizeImage_994640_9.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8de89558-ac38-49bb-b103-3219f4c9a3c2_characterizeImage_994640_9.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3a470598-b014-4d52-877c-81c521a11a33_isr_994640_9.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3a470598-b014-4d52-877c-81c521a11a33_isr_994640_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:54 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 787099d21d892671990946f0f6fdbf46 with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash c11612ac14945b314471c7158361c719 parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/96d247e5-f9a8-4295-a2bb-0a23334a25a2_characterizeImage_994639_20.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/96d247e5-f9a8-4295-a2bb-0a23334a25a2_characterizeImage_994639_20.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/10e3ef23-f34f-4199-8832-664c90ea80a8_isr_994639_20.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/10e3ef23-f34f-4199-8832-664c90ea80a8_isr_994639_20.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c768b3ca574e9d59545e494ae699cbf8 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash 50da3ee24b0477a2a11b64289d985a4b parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6507100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f4fecbb2-672f-4d9b-9cbc-c6b2b3e40dd0_characterizeImage_994639_41.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f4fecbb2-672f-4d9b-9cbc-c6b2b3e40dd0_characterizeImage_994639_41.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/fd00cde7-abca-42e7-9e98-6d8037b95590_isr_994639_41.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/fd00cde7-abca-42e7-9e98-6d8037b95590_isr_994639_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:54 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b14f776a238b544c93db30c59ce15b74 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 6e7e0228b5ea04e4536265d49dac1f58 parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6506e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/337d06e0-cce3-4d82-b88f-4d1009382e22_characterizeImage_994638_14.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/337d06e0-cce3-4d82-b88f-4d1009382e22_characterizeImage_994638_14.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/bc83f787-850d-4ada-8318-458bbf9c3601_isr_994638_14.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/bc83f787-850d-4ada-8318-458bbf9c3601_isr_994638_14.stderr parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acdb21148eba380cf32a858219f5ace0 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash 3ba585bc83ad94d73b0fea1b74a51f35 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/129c75af-580a-4dfb-87c8-5ba3ad49522b_characterizeImage_994767_17.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/129c75af-580a-4dfb-87c8-5ba3ad49522b_characterizeImage_994767_17.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/86b68bef-2b74-4d19-8324-810056343420_isr_994767_17.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/86b68bef-2b74-4d19-8324-810056343420_isr_994767_17.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17755b3949ed5b1ba194ba819142e3aa with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash 2722ed0b3a55f2536afc0b1cdacfd62d parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6506ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8e4ee58f-eca9-4187-8bd4-b273d5bfbf20_characterizeImage_994765_22.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8e4ee58f-eca9-4187-8bd4-b273d5bfbf20_characterizeImage_994765_22.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/85b27f1c-550e-4b7f-b954-30ea1817b8cf_isr_994765_22.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/85b27f1c-550e-4b7f-b954-30ea1817b8cf_isr_994765_22.stderr parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aca1adb99b46755008c8fc9ffb910af3 with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash d2cbaa5f0a353ac358624401a1bd3d29 parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/00587d06-a18c-482a-a772-ad5ad4c442f9_characterizeImage_994766_22.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/00587d06-a18c-482a-a772-ad5ad4c442f9_characterizeImage_994766_22.stderr parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/28e867cf-1ac8-40b2-9684-e420957ae04a_isr_994766_22.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/28e867cf-1ac8-40b2-9684-e420957ae04a_isr_994766_22.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c247927934045e1e2ff03f81062f8131 with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 3be0795ec61b3fe6265df1009fc92b99 parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/da68c327-1ab5-4114-9bf7-05234681ee4e_characterizeImage_994640_25.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/da68c327-1ab5-4114-9bf7-05234681ee4e_characterizeImage_994640_25.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/530c85d5-64c4-48c9-8e74-8424bf0b06b1_isr_994640_25.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/530c85d5-64c4-48c9-8e74-8424bf0b06b1_isr_994640_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:54 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f795fe2c4d8f91fcb2dfe1d4abcfa8c5 with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 7e7eb8ff3f3e039542466608223b367e parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/6a1eaef0-ee48-4ee4-bc6e-a36ef201d0f3_characterizeImage_994638_20.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/6a1eaef0-ee48-4ee4-bc6e-a36ef201d0f3_characterizeImage_994638_20.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6b946787-dae8-4eee-9042-ae63f0cdd54c_isr_994638_20.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6b946787-dae8-4eee-9042-ae63f0cdd54c_isr_994638_20.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 483d4f32517047e4887427b40d6c4146 with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash 241d60595810290fb1937d2bb4aab432 parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6506a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ef02bd2f-2ec4-451b-b142-77bd4b4499d8_characterizeImage_994640_6.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ef02bd2f-2ec4-451b-b142-77bd4b4499d8_characterizeImage_994640_6.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/da849c32-9bbd-427b-a3bf-28a5eb7158da_isr_994640_6.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/da849c32-9bbd-427b-a3bf-28a5eb7158da_isr_994640_6.stderr parsl.dataflow.dflow INFO: Task 963 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ac374fb6d6ae3c86e5dae6633c9c6c1 with result from task 963 parsl.dataflow.dflow DEBUG: Task 963 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 964 has memoization hash 5a2bca73719045d51e1f4c8a49d0cb59 parsl.dataflow.memoization INFO: Task 964 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 964 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/97d1b19d-dbf8-4030-9ffb-3b8514f21233_characterizeImage_994766_45.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/97d1b19d-dbf8-4030-9ffb-3b8514f21233_characterizeImage_994766_45.stderr parsl.dataflow.dflow INFO: Standard output for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/66e9cbe2-3aa9-4f6d-9d84-603d26de4b8a_isr_994766_45.stdout parsl.dataflow.dflow INFO: Standard error for task 963 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/66e9cbe2-3aa9-4f6d-9d84-603d26de4b8a_isr_994766_45.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee6a8d207743f5b2ff649533a7a6e0bb with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash ee3298ff03b786aaff061c9acda1a351 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/39b7ca8a-7104-48c3-a9db-32d081257021_characterizeImage_994766_50.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/39b7ca8a-7104-48c3-a9db-32d081257021_characterizeImage_994766_50.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/add5ee37-2ba1-4909-baa3-035a13153a29_isr_994766_50.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/add5ee37-2ba1-4909-baa3-035a13153a29_isr_994766_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:54 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1014 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8298c61f4d2f02814c5ab560f010b996 with result from task 1014 parsl.dataflow.dflow DEBUG: Task 1014 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1015 has memoization hash 790781215f24cc6fd2a80c64db94d6ef parsl.dataflow.memoization INFO: Task 1015 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1015 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/856ca72b-949e-4c20-b533-78fb7f41c967_characterizeImage_994767_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/856ca72b-949e-4c20-b533-78fb7f41c967_characterizeImage_994767_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/fb3c1144-fd99-4eea-bd98-7bcd70c1ffe4_isr_994767_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1014 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/fb3c1144-fd99-4eea-bd98-7bcd70c1ffe4_isr_994767_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:54 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c165332bd818ef09689e45b251da1752 with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash ef45d256c395b9d9a21dbc949ea11b12 parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/00f6fde6-c3a2-4b5e-b493-bf1e07db9015_characterizeImage_994638_10.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/00f6fde6-c3a2-4b5e-b493-bf1e07db9015_characterizeImage_994638_10.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7aa9d160-70bb-42e8-a6af-e8e58e2de5ae_isr_994638_10.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/7aa9d160-70bb-42e8-a6af-e8e58e2de5ae_isr_994638_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:55 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdfed551a071688a485f338c8701192b with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash eb6f60bc529cbf48b8786ac9011c1140 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/417a6a62-9c0c-4068-8e76-4aca7a28e2db_characterizeImage_994640_31.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/417a6a62-9c0c-4068-8e76-4aca7a28e2db_characterizeImage_994640_31.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d713747a-76c5-46c9-aa1d-f1bb546b5e40_isr_994640_31.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d713747a-76c5-46c9-aa1d-f1bb546b5e40_isr_994640_31.stderr parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 267fe833b7587bc86428fc9b3fc33714 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash 8108ec6e9f92cc5dd3e296c854d95ade parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6487a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/9634f31b-2128-4f1e-b052-dc0d6a1aef20_characterizeImage_994639_33.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/9634f31b-2128-4f1e-b052-dc0d6a1aef20_characterizeImage_994639_33.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/460b9fbd-13b9-44b7-accf-9cefd44fc6d9_isr_994639_33.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/460b9fbd-13b9-44b7-accf-9cefd44fc6d9_isr_994639_33.stderr parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de948deaeda3e7f8125e59336b477008 with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 60d33a40bb748dfef94b7e42c5526228 parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65077e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/216637a7-c158-43dd-8f69-4c67bf086769_characterizeImage_994640_8.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/216637a7-c158-43dd-8f69-4c67bf086769_characterizeImage_994640_8.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/27e3bd17-1ef1-48cd-9193-1f7086eef65e_isr_994640_8.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/27e3bd17-1ef1-48cd-9193-1f7086eef65e_isr_994640_8.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acd0accb9ebee11f54541f650a1a6f76 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash 2c52e5172887e423b8df29b69212055e parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6652f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/c6e42758-3864-4a66-9cf6-2be6955d4ebb_characterizeImage_994638_6.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/c6e42758-3864-4a66-9cf6-2be6955d4ebb_characterizeImage_994638_6.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/611ea10c-28b6-4ca5-9bf0-3595b8a716d4_isr_994638_6.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/611ea10c-28b6-4ca5-9bf0-3595b8a716d4_isr_994638_6.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3fd93d92f93e96b476573f9406eeb4d with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash e8f85394bb2d900f11fda7c174b1bed4 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/cb26b362-d62c-41c3-8eb9-1d62c76516e1_characterizeImage_994640_44.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/cb26b362-d62c-41c3-8eb9-1d62c76516e1_characterizeImage_994640_44.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/46f14f2b-234b-4439-9720-7d1d3180f0b6_isr_994640_44.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/46f14f2b-234b-4439-9720-7d1d3180f0b6_isr_994640_44.stderr parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9499d0c5e2c0df2990f12d8c6c706b73 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 232dd4caca653cce34611363ea030ee8 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b3fec195-97e4-454f-bd4f-b832115922b5_characterizeImage_994765_39.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b3fec195-97e4-454f-bd4f-b832115922b5_characterizeImage_994765_39.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b60b0587-3042-4463-9b85-87e77d07b115_isr_994765_39.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b60b0587-3042-4463-9b85-87e77d07b115_isr_994765_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:55 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3f40a2f93c2c1a321b2117f0ff719a8 with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash 292137538f6097e1318c1b1c3044dcc4 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65056c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/2ab94977-dc99-475f-8deb-ad07c8d25913_characterizeImage_994639_54.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/2ab94977-dc99-475f-8deb-ad07c8d25913_characterizeImage_994639_54.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cd4e75c5-4733-4945-b8dd-9124ba186274_isr_994639_54.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cd4e75c5-4733-4945-b8dd-9124ba186274_isr_994639_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:55 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f39f27b258516c7f431e6d0df513f262 with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash 3b55bbd5c5a663b1a7d75dc4fc38c189 parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d1b3d1bc-57fe-46ac-8579-17c081735629_characterizeImage_994639_50.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d1b3d1bc-57fe-46ac-8579-17c081735629_characterizeImage_994639_50.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ac66bcc3-130c-46b5-98a0-ee1f55b0a6ff_isr_994639_50.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ac66bcc3-130c-46b5-98a0-ee1f55b0a6ff_isr_994639_50.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dafae5ee7a7587be0031b9ab527bd7b with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 25789bab875958374445531eb7039e1e parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/4cfca11e-e4ed-4449-9e1b-4882ca7dfdee_characterizeImage_994639_57.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/4cfca11e-e4ed-4449-9e1b-4882ca7dfdee_characterizeImage_994639_57.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/7933944d-eaaa-4932-93c1-f2b5dddc1b60_isr_994639_57.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/7933944d-eaaa-4932-93c1-f2b5dddc1b60_isr_994639_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:55 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1017 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e59865e4e8846d16822036f814c2796c with result from task 1017 parsl.dataflow.dflow DEBUG: Task 1017 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1018 has memoization hash ee74d0b1fb8a825f9259e9cf229e5473 parsl.dataflow.memoization INFO: Task 1018 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1018 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f3b92211-1f61-40bc-a47c-f96136499daf_characterizeImage_994639_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f3b92211-1f61-40bc-a47c-f96136499daf_characterizeImage_994639_29.stderr parsl.dataflow.dflow INFO: Standard output for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/c213fe68-637d-40dc-85fa-663af5f13067_isr_994639_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1017 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/c213fe68-637d-40dc-85fa-663af5f13067_isr_994639_29.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3741504585ce19ba755ecef67f18fcee with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 6cf324825a590b7b543c84bfa5f78a6f parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65071a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/103a32e2-ebdb-47ed-97d4-f0644d2c6dff_characterizeImage_994766_25.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/103a32e2-ebdb-47ed-97d4-f0644d2c6dff_characterizeImage_994766_25.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d8405de3-8a8a-4dd8-a589-aa3a6616c5f9_isr_994766_25.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d8405de3-8a8a-4dd8-a589-aa3a6616c5f9_isr_994766_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:55 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 114657199e763934f3f32d488c097ef5 with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash 58191daddb985d30a1cd693aa0077dda parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65074c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/c2228bed-f33e-450a-ad05-93d422e19e07_characterizeImage_994638_24.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/c2228bed-f33e-450a-ad05-93d422e19e07_characterizeImage_994638_24.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f3b4a8d3-66af-4607-864f-f3f76f52845c_isr_994638_24.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f3b4a8d3-66af-4607-864f-f3f76f52845c_isr_994638_24.stderr parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84dab25477fda71720f6062b9c7b9c6d with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 663274d2480097cdb65e729680deb7a9 parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6507f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a9092632-1ce3-47ce-be45-334f449f714f_characterizeImage_994638_16.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a9092632-1ce3-47ce-be45-334f449f714f_characterizeImage_994638_16.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/0fe5f965-7266-4ce3-a9fd-be5965827da6_isr_994638_16.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/0fe5f965-7266-4ce3-a9fd-be5965827da6_isr_994638_16.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94754bbfb84ffa50d2af3b6b3571a87d with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 402a137e3ca1151e17c2af66ce85fe74 parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/6c707e79-3404-48fd-a09f-f9effae741bc_characterizeImage_994766_39.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/6c707e79-3404-48fd-a09f-f9effae741bc_characterizeImage_994766_39.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/a973a03d-64d3-419f-9ba2-c4b2b0b1578e_isr_994766_39.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/a973a03d-64d3-419f-9ba2-c4b2b0b1578e_isr_994766_39.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fb955b834b9391783b98196d198c3c4 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash d5539341f89a323ba9c225fa051caa6c parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65053a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/752867fe-b703-4718-9716-acd73ea20183_characterizeImage_994765_38.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/752867fe-b703-4718-9716-acd73ea20183_characterizeImage_994765_38.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/eb7fcd46-abb2-4718-8923-58f7bf8b4bd6_isr_994765_38.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/eb7fcd46-abb2-4718-8923-58f7bf8b4bd6_isr_994765_38.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37977e7397f49f8bb5d22d45cfc53821 with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash c35db0320d6b5142b6ad4c0318e8b2c6 parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/dca4f1ba-4451-4053-bf79-1635a390d6f8_characterizeImage_994767_5.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/dca4f1ba-4451-4053-bf79-1635a390d6f8_characterizeImage_994767_5.stderr parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/2d0706db-7600-4b0e-a8f6-638ed3bb35c0_isr_994767_5.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/2d0706db-7600-4b0e-a8f6-638ed3bb35c0_isr_994767_5.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00cea9b389115e2273cdaa628532deaf with result from task 858 parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Task 859 has memoization hash fdca1e100d0105ddd6bbc8f070b2c668 parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:55 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9b728f80-0619-46c6-a6b1-555cea43eeeb_characterizeImage_994638_9.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9b728f80-0619-46c6-a6b1-555cea43eeeb_characterizeImage_994638_9.stderr parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d282c0c5-b182-4cf0-9b36-fd74111ba68e_isr_994638_9.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d282c0c5-b182-4cf0-9b36-fd74111ba68e_isr_994638_9.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37c63c98b37067feff3d9aa2fd2bc9b4 with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 36c10c0ec012e42ca978f0c0d3461752 parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5cad8192-20c4-4049-aebd-8512698ddc0a_characterizeImage_994766_40.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5cad8192-20c4-4049-aebd-8512698ddc0a_characterizeImage_994766_40.stderr parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/12aed698-4e9b-40c7-a6a6-b90929512803_isr_994766_40.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/12aed698-4e9b-40c7-a6a6-b90929512803_isr_994766_40.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fea4a89d6aeb204b3b3322c1fd2d7fa4 with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash 620283d0accd7b35cbdeded63eb0fd47 parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ff61e7a2-5153-4486-a98e-a58a611daf1c_characterizeImage_994639_35.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ff61e7a2-5153-4486-a98e-a58a611daf1c_characterizeImage_994639_35.stderr parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cf5619b3-fad7-4952-9e74-5846dffd537a_isr_994639_35.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/cf5619b3-fad7-4952-9e74-5846dffd537a_isr_994639_35.stderr parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed391d4dcee4043b215e62f0c1025315 with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 28af9932da73fcf00298d3f022bf22ab parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/d2970fc3-e43d-4e47-bbf7-22c5256eeca5_characterizeImage_994766_62.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/d2970fc3-e43d-4e47-bbf7-22c5256eeca5_characterizeImage_994766_62.stderr parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/6e60b9dc-debe-446d-a2dd-dc1b39247ab0_isr_994766_62.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/6e60b9dc-debe-446d-a2dd-dc1b39247ab0_isr_994766_62.stderr parsl.dataflow.dflow INFO: Task 1098 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d57ebab56dcfc9eadf7f86484b7c1a94 with result from task 1098 parsl.dataflow.dflow DEBUG: Task 1098 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1099 has memoization hash 514a14d654c4984e5c4dac0f4efc303a parsl.dataflow.memoization INFO: Task 1099 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1099 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/78fb25cf-d016-43dc-9c40-87aaba915688_characterizeImage_994640_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/78fb25cf-d016-43dc-9c40-87aaba915688_characterizeImage_994640_36.stderr parsl.dataflow.dflow INFO: Standard output for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/69402ed9-fc8a-4a2c-af8b-6684d3cfae58_isr_994640_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1098 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/69402ed9-fc8a-4a2c-af8b-6684d3cfae58_isr_994640_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:55 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb8bb71d1a114a4568725e89ff0e1f40 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash 45b44074824b7adab8429ecb9b182877 parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/996e02ab-de4d-4835-945b-226f07041bb3_characterizeImage_994765_7.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/996e02ab-de4d-4835-945b-226f07041bb3_characterizeImage_994765_7.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/15b2c357-e0b3-462c-880a-4f374ff1da39_isr_994765_7.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/15b2c357-e0b3-462c-880a-4f374ff1da39_isr_994765_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:56 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2676726aca938a638a2cfe8c00e252c0 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash dda74284a743c8a3433b9615b9699a9c parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5c5420e7-7859-43df-9f20-0c69df7d1111_characterizeImage_994766_58.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5c5420e7-7859-43df-9f20-0c69df7d1111_characterizeImage_994766_58.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/6af1e63c-3f68-4792-9c97-d31ea6739f64_isr_994766_58.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/6af1e63c-3f68-4792-9c97-d31ea6739f64_isr_994766_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:56 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 264bab75030ae69c73e02cd7916d6afc with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 00b7c70247dd50cc52372ae2dcc5b09c parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3db63ef5-7b61-49c4-a4d1-75af970dba37_characterizeImage_994639_36.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3db63ef5-7b61-49c4-a4d1-75af970dba37_characterizeImage_994639_36.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/40dc4367-1d2a-4b02-b3ed-76f18b4052ff_isr_994639_36.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/40dc4367-1d2a-4b02-b3ed-76f18b4052ff_isr_994639_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:56 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da90b260bb5af5ea1d88758d0fdfb28d with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash a5fb0b28229eb91d8e5f694eed23a47e parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64862a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ab8b43d2-894a-4d0b-8430-d755ad1e1ff3_characterizeImage_994638_54.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ab8b43d2-894a-4d0b-8430-d755ad1e1ff3_characterizeImage_994638_54.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6f66111e-1fbe-4297-8fa1-f7e296b5c6ca_isr_994638_54.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6f66111e-1fbe-4297-8fa1-f7e296b5c6ca_isr_994638_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:56 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77f289653ede9b4652bad18191e72d08 with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash b0bfe20ddb8a7c4d2ea5dfb3707869aa parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6507b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/558ebeb7-e54a-47f4-9e8f-1ede5c900846_characterizeImage_994640_59.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/558ebeb7-e54a-47f4-9e8f-1ede5c900846_characterizeImage_994640_59.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5a37a655-6382-4314-b6d1-ff4ac4682415_isr_994640_59.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5a37a655-6382-4314-b6d1-ff4ac4682415_isr_994640_59.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3701874955241b129355308b071b2510 with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 41445637675acf1afe251e828c4dbd0d parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/678b144c-e391-4f17-aff7-92cfd6164e39_characterizeImage_994638_8.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/678b144c-e391-4f17-aff7-92cfd6164e39_characterizeImage_994638_8.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/bf529c42-cdef-4021-971b-3a7fc35629db_isr_994638_8.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/bf529c42-cdef-4021-971b-3a7fc35629db_isr_994638_8.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bb0c6b0c6598da8e0094d0cbe9919b4 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 7c5c62a90fe0834d1e80c2fcc7ca8c21 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66520c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/089de6e3-d293-4286-846f-8e5da18d086d_characterizeImage_994640_34.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/089de6e3-d293-4286-846f-8e5da18d086d_characterizeImage_994640_34.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5cb35976-1065-4111-9a7c-4ff1de95ca7e_isr_994640_34.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5cb35976-1065-4111-9a7c-4ff1de95ca7e_isr_994640_34.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 253e78290b8dcf50d8494ad539b0d724 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 23d4a9df9e6a5a10eceeeac86166d02c parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/9fb224e5-2b04-459f-b2d1-5128ca4d35ed_characterizeImage_994765_41.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/9fb224e5-2b04-459f-b2d1-5128ca4d35ed_characterizeImage_994765_41.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/222e2ccf-a16f-4f6d-a78d-46095046f79a_isr_994765_41.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/222e2ccf-a16f-4f6d-a78d-46095046f79a_isr_994765_41.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44e4a02c4ba1dec3f649d3e15d3f5914 with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 64a0c82596fef9f07384d780ed6d2f32 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6652160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/96212b4c-6269-4f4e-a005-bb5a41f6be5b_characterizeImage_994640_35.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/96212b4c-6269-4f4e-a005-bb5a41f6be5b_characterizeImage_994640_35.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f773fce6-1aa2-4676-9519-1ecc1f386f2d_isr_994640_35.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f773fce6-1aa2-4676-9519-1ecc1f386f2d_isr_994640_35.stderr parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb3aa121c03ee57270812d9e25a19710 with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 506ad8943533ad2154b526b702acd362 parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/9ac183ae-ea5d-4800-88c6-937708a5521c_characterizeImage_994766_29.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/9ac183ae-ea5d-4800-88c6-937708a5521c_characterizeImage_994766_29.stderr parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/283c171a-e1dc-4492-b1d5-485d7c3101ae_isr_994766_29.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/283c171a-e1dc-4492-b1d5-485d7c3101ae_isr_994766_29.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e41d16c666ddbe7c0468e4c495f32dba with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash eba8bffb09159de60fda4d7d7029c265 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/98637530-2b75-4eb1-ae70-f92fc430b7df_characterizeImage_994767_32.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/98637530-2b75-4eb1-ae70-f92fc430b7df_characterizeImage_994767_32.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9d9d2f3c-0d15-44db-8e7a-6f93015d141c_isr_994767_32.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9d9d2f3c-0d15-44db-8e7a-6f93015d141c_isr_994767_32.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf554339a0701b025fa6d92bbb6e3630 with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash e60d0d4b3afe3ea7d633f020c8cf4f60 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/323bcf3e-aa70-4e09-874b-a817e16f6c22_characterizeImage_994639_5.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/323bcf3e-aa70-4e09-874b-a817e16f6c22_characterizeImage_994639_5.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/50e6c21a-e104-43ba-a467-c29c98f76533_isr_994639_5.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/50e6c21a-e104-43ba-a467-c29c98f76533_isr_994639_5.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf688d871492d2407cf6e20f6a10bfb0 with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash 046cdc6186c5b9bc3dd3e98c67b3c5a8 parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ecbef09d-7d5e-4cec-8b89-6a566a8c9d28_characterizeImage_994638_21.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ecbef09d-7d5e-4cec-8b89-6a566a8c9d28_characterizeImage_994638_21.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/4fd0c220-087b-4a04-8a6b-5b651257c4fc_isr_994638_21.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/4fd0c220-087b-4a04-8a6b-5b651257c4fc_isr_994638_21.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad3f93ca59e32d6bad775e7195f7d1ae with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash e1434d3b3a2f69ae4886aef112c0abc1 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/6026d1e9-c50c-4b5b-9c83-79f1fd1bd6e8_characterizeImage_994765_40.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/6026d1e9-c50c-4b5b-9c83-79f1fd1bd6e8_characterizeImage_994765_40.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/e08f7824-71f8-4d1f-b16e-854c4a1d9bfe_isr_994765_40.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/e08f7824-71f8-4d1f-b16e-854c4a1d9bfe_isr_994765_40.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 748080e00b4e190ac2d9a3fa11c08819 with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash edba5b927df503d246a31e59dbcdc161 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:56 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/faa468f8-1d91-496b-908d-135642a00b58_characterizeImage_994765_10.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/faa468f8-1d91-496b-908d-135642a00b58_characterizeImage_994765_10.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/7f99a35b-fc65-43c1-87c2-ee5f75942eed_isr_994765_10.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/7f99a35b-fc65-43c1-87c2-ee5f75942eed_isr_994765_10.stderr parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eae29897d3aa437b45608d154699d6f1 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 3a51dea19e206eea545bbce58278c48b parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/fa93fab6-e60a-4b8f-b68b-e0aea60292ad_characterizeImage_994765_6.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/fa93fab6-e60a-4b8f-b68b-e0aea60292ad_characterizeImage_994765_6.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/217977e2-6180-4e0c-ab5a-aeaa697f8cd0_isr_994765_6.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/217977e2-6180-4e0c-ab5a-aeaa697f8cd0_isr_994765_6.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e0b414ec58aa8a4001d9e9e7fc83e12 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 9100e25b96fede540787daf3fdfe49a9 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6487560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/cb79766a-e1d8-4355-a4f2-45b9c67c9190_characterizeImage_994638_22.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/cb79766a-e1d8-4355-a4f2-45b9c67c9190_characterizeImage_994638_22.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8fad4a9a-0f4b-45d0-9b78-790414eaad90_isr_994638_22.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8fad4a9a-0f4b-45d0-9b78-790414eaad90_isr_994638_22.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 168791fa61b44d0afd021fd9deacf140 with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 4648eda29cbf5fe25c1e1fa889691496 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8a192b74-8f48-4dd3-8d95-aafd0ee49930_characterizeImage_994765_20.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8a192b74-8f48-4dd3-8d95-aafd0ee49930_characterizeImage_994765_20.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/85cc45bd-f32a-4c04-93de-6a157381d2da_isr_994765_20.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/85cc45bd-f32a-4c04-93de-6a157381d2da_isr_994765_20.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb6c00fb53d58393dbc085e33e9d398d with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash b9c5dc142898e5671265004975b4629c parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b8661903-be6e-44eb-a3fa-a51429affe2c_characterizeImage_994765_28.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b8661903-be6e-44eb-a3fa-a51429affe2c_characterizeImage_994765_28.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/611dd07c-64ff-4593-9eb2-26150228486a_isr_994765_28.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/611dd07c-64ff-4593-9eb2-26150228486a_isr_994765_28.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60285cac25fad7af6f25c50a4c8b1335 with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash 1b981a82266165e9aab9d7416d8a3df8 parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/f8d2b92d-abf3-452e-8813-95a698a3a2a7_characterizeImage_994767_20.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/f8d2b92d-abf3-452e-8813-95a698a3a2a7_characterizeImage_994767_20.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c8363a36-05fe-482d-813d-0a37d2391b95_isr_994767_20.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/c8363a36-05fe-482d-813d-0a37d2391b95_isr_994767_20.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86fec7fe42df64f23ab3e18caf986039 with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash e4529521fd7cdb23bf1e475de3e5ff91 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2d730346-779d-4d10-967e-627f4a1889d2_characterizeImage_994766_56.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2d730346-779d-4d10-967e-627f4a1889d2_characterizeImage_994766_56.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/efaa96a5-8dec-47a6-bdad-34ef5a96e7be_isr_994766_56.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/efaa96a5-8dec-47a6-bdad-34ef5a96e7be_isr_994766_56.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c2bc0ae5bd7289d4ea2f197d1572804 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash 1871bfdea56de215cc2a0f771c1c1f57 parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2a357535-cd81-4e24-9d73-e7ef047439ba_characterizeImage_994640_50.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2a357535-cd81-4e24-9d73-e7ef047439ba_characterizeImage_994640_50.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/0aab6239-7f5f-4f11-ba42-e9e52cd824a5_isr_994640_50.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/0aab6239-7f5f-4f11-ba42-e9e52cd824a5_isr_994640_50.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63e48b4558479b293752ebf838b8ee24 with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash 02f44a2182f2f70e68bdfa27d2dbd0ac parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0f802a9c-7951-478c-951a-f41789e31a66_characterizeImage_994639_58.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0f802a9c-7951-478c-951a-f41789e31a66_characterizeImage_994639_58.stderr parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/94cf4a76-b9b8-4f45-89a5-b7ad419573e3_isr_994639_58.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/94cf4a76-b9b8-4f45-89a5-b7ad419573e3_isr_994639_58.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0d77ffce04810e6b019223f0864d2fe with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash 93b4421edcc9a2b4b0c8c45ac720e26b parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/9b6e824f-4c39-4333-a77a-3b170c7b28dc_characterizeImage_994640_7.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/9b6e824f-4c39-4333-a77a-3b170c7b28dc_characterizeImage_994640_7.stderr parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/6d54a996-ca86-4dd7-bf32-855a5ea4df71_isr_994640_7.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/6d54a996-ca86-4dd7-bf32-855a5ea4df71_isr_994640_7.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23340b95e56c4e52e29994b8f75f34aa with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash ed34ce52c4b80bb4e793cafc692ef2f1 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b76576dd-f549-45ba-b4c9-116ee2956270_characterizeImage_994640_22.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b76576dd-f549-45ba-b4c9-116ee2956270_characterizeImage_994640_22.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/23782b8c-97fe-417b-b42b-bace24f72074_isr_994640_22.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/23782b8c-97fe-417b-b42b-bace24f72074_isr_994640_22.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af92ca857ac2685fcb01a1e064139d71 with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash d557eadb3d885f54b5af0de2e14776ce parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64879c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/69d498d3-aabe-4186-935d-1197a9345c9d_characterizeImage_994767_46.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/69d498d3-aabe-4186-935d-1197a9345c9d_characterizeImage_994767_46.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9860b7ef-0a2f-4fd1-a60e-a679cca60d77_isr_994767_46.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9860b7ef-0a2f-4fd1-a60e-a679cca60d77_isr_994767_46.stderr parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7faadff62b5057329d37b0b821e8943f with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash 513953b5b751ea11f04d65e992723e9f parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6487600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8f0ef8bb-3959-483b-9c98-04c78c7577e6_characterizeImage_994640_17.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8f0ef8bb-3959-483b-9c98-04c78c7577e6_characterizeImage_994640_17.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e5e457fa-6bba-435c-8346-fcb85e3be101_isr_994640_17.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e5e457fa-6bba-435c-8346-fcb85e3be101_isr_994640_17.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd3290ebcb2fd2fe06501e34fb1f485d with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 88269b69217f24d45853b35b348d2074 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/dca9f8ac-6826-4305-90ba-12c8c4fd47a1_characterizeImage_994638_42.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/dca9f8ac-6826-4305-90ba-12c8c4fd47a1_characterizeImage_994638_42.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/88f7f43f-a12e-4f53-9182-7aac0b44b170_isr_994638_42.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/88f7f43f-a12e-4f53-9182-7aac0b44b170_isr_994638_42.stderr parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60bb53bc2ec01295f39ecdf4996376cd with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 5dab2aa1155fef4d34a98413808e9dec parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2bd03605-3947-4e60-a262-eee9e5c81f88_characterizeImage_994767_23.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2bd03605-3947-4e60-a262-eee9e5c81f88_characterizeImage_994767_23.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/be6d9d0b-5cd3-4a22-98e4-c884fbc2baef_isr_994767_23.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/be6d9d0b-5cd3-4a22-98e4-c884fbc2baef_isr_994767_23.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0b2823a6a3ba3c0514d5bc3d64ef8c3 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 13c64a1b3de4d1319ff0f48a3aceef2f parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:56 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/76b6a8b7-49c6-4424-8e48-5e6e3454ab06_characterizeImage_994765_31.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/76b6a8b7-49c6-4424-8e48-5e6e3454ab06_characterizeImage_994765_31.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/95ab9ba8-9bf3-4037-a1d2-c45b21c9c162_isr_994765_31.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/95ab9ba8-9bf3-4037-a1d2-c45b21c9c162_isr_994765_31.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ee8ec9229d88bf3b0e2773a2d991a25 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 1cea13fdc7c96e928d4dae834a97cca0 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/092009d3-b0f8-4d69-98a1-512af2898b08_characterizeImage_994766_35.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/092009d3-b0f8-4d69-98a1-512af2898b08_characterizeImage_994766_35.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2a407662-e142-41bc-9e8f-bea15f982d55_isr_994766_35.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2a407662-e142-41bc-9e8f-bea15f982d55_isr_994766_35.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f97a94308b454d7a2294ebaff54a085 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 07404ad2920e0a26a3b12818b224f6f6 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/fc07e084-d24c-4739-bb51-dc071799d400_characterizeImage_994766_13.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/fc07e084-d24c-4739-bb51-dc071799d400_characterizeImage_994766_13.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b4e44f8f-4e91-40ef-8652-0c4ba03e1399_isr_994766_13.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b4e44f8f-4e91-40ef-8652-0c4ba03e1399_isr_994766_13.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 537540529a9511dcbdf0bfb6e5f092d3 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash bffd49c32dc79d8662c5cf6c309af930 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/d617abdf-0be1-4635-a57b-f0e822b8807d_characterizeImage_994766_37.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/d617abdf-0be1-4635-a57b-f0e822b8807d_characterizeImage_994766_37.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/555b860b-59e1-4e42-9d2c-9a86d1d6922a_isr_994766_37.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/555b860b-59e1-4e42-9d2c-9a86d1d6922a_isr_994766_37.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98aaaa1b99e46c4fb797f10a5f8127ba with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 5643bc25f63cd0c72aafc4071d600d32 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6487ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/85cc91ff-4723-4451-aa26-b0e1c5557cfd_characterizeImage_994765_37.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/85cc91ff-4723-4451-aa26-b0e1c5557cfd_characterizeImage_994765_37.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/c2cda851-efcf-4612-addc-684f8200eb67_isr_994765_37.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/c2cda851-efcf-4612-addc-684f8200eb67_isr_994765_37.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a7e1c1855a51918802346b097d9b3fb with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 5d1e8cd34c466b29d174ff178e2d9ad7 parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/241fbe34-1149-4d9c-ae4f-473dddd7a7f8_characterizeImage_994767_43.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/241fbe34-1149-4d9c-ae4f-473dddd7a7f8_characterizeImage_994767_43.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8a5b24aa-03a3-42d1-91c0-fa5c8d8e9499_isr_994767_43.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8a5b24aa-03a3-42d1-91c0-fa5c8d8e9499_isr_994767_43.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67e82d121a297b07170d50e96602d881 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 0d1d5f1bd415964a744db117f8a0b35c parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/a2b89b58-c51e-4250-86a6-009a90d37b56_characterizeImage_994767_9.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/a2b89b58-c51e-4250-86a6-009a90d37b56_characterizeImage_994767_9.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ce822876-e0eb-4a86-b564-393175aea6c2_isr_994767_9.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ce822876-e0eb-4a86-b564-393175aea6c2_isr_994767_9.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6dcbedc2885f48d36db61d0d37c124a with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 5ce190f5a02ed1dbd22f05b3569a7410 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8173a87e-aa6e-4579-baff-6168317b540a_characterizeImage_994639_46.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8173a87e-aa6e-4579-baff-6168317b540a_characterizeImage_994639_46.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/b3f99e42-d278-4f1c-a5e0-16c789af7a3d_isr_994639_46.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/b3f99e42-d278-4f1c-a5e0-16c789af7a3d_isr_994639_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:57 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5514884108304050f629d23386d0873 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 7729559f48603683b4e731248a7ffa62 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2ff4ad02-3989-4423-ad59-71b3cfe7a256_characterizeImage_994767_54.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2ff4ad02-3989-4423-ad59-71b3cfe7a256_characterizeImage_994767_54.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/17c9b672-695e-4e94-9666-a57ce13ac6a7_isr_994767_54.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/17c9b672-695e-4e94-9666-a57ce13ac6a7_isr_994767_54.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 76 try 0 failed parsl.dataflow.dflow INFO: Task 76 marked for retry parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a2701b9f-845a-43f3-bafd-8d03975dedc0_characterizeImage_994765_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a2701b9f-845a-43f3-bafd-8d03975dedc0_characterizeImage_994765_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 0c480b101e15696aa976ad188dcde24e parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 1 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a2701b9f-845a-43f3-bafd-8d03975dedc0_characterizeImage_994765_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a2701b9f-845a-43f3-bafd-8d03975dedc0_characterizeImage_994765_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:57 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:57 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df58a6ac312a0e77c988f97b888b719e with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 97483f7b040ae9d90ab5013f1a62fc00 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/919cf315-3584-4e95-8b2e-7fd38e8a2282_characterizeImage_994767_36.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/919cf315-3584-4e95-8b2e-7fd38e8a2282_characterizeImage_994767_36.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/0638be76-00f8-4130-bcbe-c7c58da51997_isr_994767_36.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/0638be76-00f8-4130-bcbe-c7c58da51997_isr_994767_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:13:58 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82948838639926b46abd9ef46644dc3a with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash c106e286ad3089d7cc3856e0f8157d33 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/de1b72a2-fa6c-463a-8613-b50d50d8041f_characterizeImage_994767_21.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/de1b72a2-fa6c-463a-8613-b50d50d8041f_characterizeImage_994767_21.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/f2349455-c72c-4919-9069-df159487babe_isr_994767_21.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/f2349455-c72c-4919-9069-df159487babe_isr_994767_21.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa1eb39e6f9222aec6474bad3afbcf16 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 2f50db04d162e117d5e8dd49a7489313 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8ef89b07-43ff-48f4-993b-ad655ad25f11_characterizeImage_994640_37.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8ef89b07-43ff-48f4-993b-ad655ad25f11_characterizeImage_994640_37.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c9571dda-dcae-4ea9-b966-9de27dfeddb3_isr_994640_37.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c9571dda-dcae-4ea9-b966-9de27dfeddb3_isr_994640_37.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44f0a978bd772e29f2b2ddb3519480fc with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash 28a9540799c981f74563ac7189b86f0c parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/87acb1a4-78ef-457b-be18-b232481d00b8_characterizeImage_994639_45.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/87acb1a4-78ef-457b-be18-b232481d00b8_characterizeImage_994639_45.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/8812509c-3e56-4352-b3a2-99e1d13f8301_isr_994639_45.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/8812509c-3e56-4352-b3a2-99e1d13f8301_isr_994639_45.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a7838e7313d1f6bf25bfc71031f1685 with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 8231195fca1bbee2730752bf628bcce5 parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63918a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ca86d385-d072-430a-bcb6-db791875473f_characterizeImage_994765_44.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ca86d385-d072-430a-bcb6-db791875473f_characterizeImage_994765_44.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87084e17-5e27-4858-b035-b7b56ddbd2c8_isr_994765_44.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87084e17-5e27-4858-b035-b7b56ddbd2c8_isr_994765_44.stderr parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0868d867527eb9ea61dff50b54877af with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 1f4550b36df7e5491cf9d6286cfa53f2 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63905e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0fd6d13c-0b81-4963-874a-3a3aac492273_characterizeImage_994767_28.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0fd6d13c-0b81-4963-874a-3a3aac492273_characterizeImage_994767_28.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7c695369-0353-4861-9f68-f5de52f83a50_isr_994767_28.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/7c695369-0353-4861-9f68-f5de52f83a50_isr_994767_28.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea158f8c40cd65cc13f6e90a76cd6975 with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash c78f372de130ce2941b216d1b69e331e parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63920c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/465073c6-35cb-4523-82f6-76f5ddfee818_characterizeImage_994766_27.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/465073c6-35cb-4523-82f6-76f5ddfee818_characterizeImage_994766_27.stderr parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4342d7cf-5df4-4459-8022-80675a168afd_isr_994766_27.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4342d7cf-5df4-4459-8022-80675a168afd_isr_994766_27.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fca5a72c07152e60b888a9569891f6f0 with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash cf7915937e788cc578eb1b6a5e87ef5c parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/af7d583a-8850-44fb-a07d-c3357e0bf9b8_characterizeImage_994638_25.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/af7d583a-8850-44fb-a07d-c3357e0bf9b8_characterizeImage_994638_25.stderr parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/5b9c8767-fabb-443d-988c-c72202ec7be4_isr_994638_25.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/5b9c8767-fabb-443d-988c-c72202ec7be4_isr_994638_25.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c36f1bf8cbb334762ccc06972ac3ac92 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 71daa22c9fd812354ba2f1d1ce577480 parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6392a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b93f87ee-ba60-4cf7-8830-8ec24c59d345_characterizeImage_994640_33.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b93f87ee-ba60-4cf7-8830-8ec24c59d345_characterizeImage_994640_33.stderr parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/6ddbe8b7-db37-4ccb-b75e-7f428a525f42_isr_994640_33.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/6ddbe8b7-db37-4ccb-b75e-7f428a525f42_isr_994640_33.stderr parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b62d5d58ba3e3f439dd63eb0a6a0843 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash f25c4833b446544a814853b3bd72e81e parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63932e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/cfe2560d-6f26-499c-9626-fb1c2ab00b85_characterizeImage_994765_61.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/cfe2560d-6f26-499c-9626-fb1c2ab00b85_characterizeImage_994765_61.stderr parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/71b0e97a-21c2-474e-80d9-38c421d2734c_isr_994765_61.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/71b0e97a-21c2-474e-80d9-38c421d2734c_isr_994765_61.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0bf12c855e349cad1a553fb9c1500ac with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash cfacafd185aa926f78c1e5ba47042243 parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3b8872a1-af2c-42c2-a790-854423945540_characterizeImage_994639_1.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3b8872a1-af2c-42c2-a790-854423945540_characterizeImage_994639_1.stderr parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/2266382e-5d29-4f33-91d0-a96fdbc73c13_isr_994639_1.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/2266382e-5d29-4f33-91d0-a96fdbc73c13_isr_994639_1.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f86cc135febc98117f1c2523b3a9d6c with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash 131c4639e08a796bc4804806a91da93b parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6392d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/e5061510-ea4d-463b-a166-d283b5bf511c_characterizeImage_994767_42.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/e5061510-ea4d-463b-a166-d283b5bf511c_characterizeImage_994767_42.stderr parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cf705f0d-8331-4fba-91f9-29b9c66737b0_isr_994767_42.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/cf705f0d-8331-4fba-91f9-29b9c66737b0_isr_994767_42.stderr parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32f5f577428f8e537c1ff4cd43bb3f19 with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 7c7b3b80289e4ecc688dd80e42835d0d parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/620f7ba6-aca5-4121-8486-275d90b43ced_characterizeImage_994765_3.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/620f7ba6-aca5-4121-8486-275d90b43ced_characterizeImage_994765_3.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a675da93-565d-48a5-8ffe-e257fc21204e_isr_994765_3.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a675da93-565d-48a5-8ffe-e257fc21204e_isr_994765_3.stderr parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60e4cf59105d659da7f8adf3ab29ec03 with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash e7318551654b675182dd985d4d46f15f parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e50fc3a6-1172-4b31-a9af-1135983e2bd0_characterizeImage_994640_27.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e50fc3a6-1172-4b31-a9af-1135983e2bd0_characterizeImage_994640_27.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/42bcf247-aaaa-4823-8d8b-9cdf870aac6f_isr_994640_27.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/42bcf247-aaaa-4823-8d8b-9cdf870aac6f_isr_994640_27.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0536db1f10d034106269ed0cb6f65c01 with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash 1fc9dd22a508dab1f9065593c1a8305a parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/80ff2920-5805-4a16-9f9d-7c5c1906d107_characterizeImage_994640_24.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/80ff2920-5805-4a16-9f9d-7c5c1906d107_characterizeImage_994640_24.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/fb3cb49e-9392-4eeb-a4f7-4bfa46996df9_isr_994640_24.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/fb3cb49e-9392-4eeb-a4f7-4bfa46996df9_isr_994640_24.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc855050a87f524dea3e48ee13e8bd45 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash ceadf86c773a2804d734defd933a76e6 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/965702f0-5260-4305-b6ba-22be2e3be93e_characterizeImage_994765_26.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/965702f0-5260-4305-b6ba-22be2e3be93e_characterizeImage_994765_26.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/d68ca3a6-605f-4c4f-af7b-52e1d88cbaf7_isr_994765_26.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/d68ca3a6-605f-4c4f-af7b-52e1d88cbaf7_isr_994765_26.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f20aa12ecc411ebbeb00dc8d923a4c75 with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash 869eba2dea750735b50a0d16afd3e454 parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6392700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9e98434c-ca64-44ef-bb5e-634632f81af9_characterizeImage_994638_13.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9e98434c-ca64-44ef-bb5e-634632f81af9_characterizeImage_994638_13.stderr parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/0822b7b1-2e3c-4e33-a42b-9e7942f6c426_isr_994638_13.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/0822b7b1-2e3c-4e33-a42b-9e7942f6c426_isr_994638_13.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6baeecfa8e818a795dfb8fcaba865455 with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash 0456cce8c4d63c12c39f2511599037ea parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63923e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d6eefd11-0721-4ba3-90ba-80f377a0ddc4_characterizeImage_994765_48.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d6eefd11-0721-4ba3-90ba-80f377a0ddc4_characterizeImage_994765_48.stderr parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0f68dd10-2266-4009-a51f-e8fa45777583_isr_994765_48.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0f68dd10-2266-4009-a51f-e8fa45777583_isr_994765_48.stderr parsl.dataflow.dflow INFO: Task 975 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 416f0acb14554b256f8d49ea0a3eef48 with result from task 975 parsl.dataflow.dflow DEBUG: Task 975 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 976 has memoization hash 4cb0035a16fbc51b3b69f09febb9e2eb parsl.dataflow.memoization INFO: Task 976 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 976 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/7df45191-73d4-49ac-9c7b-b19f8f4d2746_characterizeImage_994767_48.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/7df45191-73d4-49ac-9c7b-b19f8f4d2746_characterizeImage_994767_48.stderr parsl.dataflow.dflow INFO: Standard output for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/64c95a55-906d-453e-9fa8-05791965c528_isr_994767_48.stdout parsl.dataflow.dflow INFO: Standard error for task 975 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/64c95a55-906d-453e-9fa8-05791965c528_isr_994767_48.stderr parsl.dataflow.dflow INFO: Task 987 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dafa6ffaecd7b47eed27815afb233356 with result from task 987 parsl.dataflow.dflow DEBUG: Task 987 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 988 has memoization hash 0ca7af1bbcbde544fc545ceca0615d38 parsl.dataflow.memoization INFO: Task 988 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 988 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/10568beb-83c2-4f24-9f8d-45ecaf803da5_characterizeImage_994638_26.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/10568beb-83c2-4f24-9f8d-45ecaf803da5_characterizeImage_994638_26.stderr parsl.dataflow.dflow INFO: Standard output for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/28f13302-e0a4-4933-8179-1585df93ada4_isr_994638_26.stdout parsl.dataflow.dflow INFO: Standard error for task 987 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/28f13302-e0a4-4933-8179-1585df93ada4_isr_994638_26.stderr parsl.dataflow.dflow INFO: Task 984 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2282474ce88e2e52d2e5a038d7eec45e with result from task 984 parsl.dataflow.dflow DEBUG: Task 984 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 985 has memoization hash dc00f28e4eaaab51346e6c2c8f8321fb parsl.dataflow.memoization INFO: Task 985 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 985 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/a435d552-5623-482b-b7b6-d30bbb8e8246_characterizeImage_994640_11.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/a435d552-5623-482b-b7b6-d30bbb8e8246_characterizeImage_994640_11.stderr parsl.dataflow.dflow INFO: Standard output for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4ab5b021-4533-4a13-a5ad-69b498e1c69a_isr_994640_11.stdout parsl.dataflow.dflow INFO: Standard error for task 984 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4ab5b021-4533-4a13-a5ad-69b498e1c69a_isr_994640_11.stderr parsl.dataflow.dflow INFO: Task 981 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f706aecdad3086d910633609906c7c18 with result from task 981 parsl.dataflow.dflow DEBUG: Task 981 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 982 has memoization hash 7541247290ed82e5243238adcf37b434 parsl.dataflow.memoization INFO: Task 982 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 982 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/b298d388-4481-4b11-86e1-1b4abb14e473_characterizeImage_994766_24.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/b298d388-4481-4b11-86e1-1b4abb14e473_characterizeImage_994766_24.stderr parsl.dataflow.dflow INFO: Standard output for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ad8607be-643f-494a-a0f4-cb87aeb0bcc7_isr_994766_24.stdout parsl.dataflow.dflow INFO: Standard error for task 981 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ad8607be-643f-494a-a0f4-cb87aeb0bcc7_isr_994766_24.stderr parsl.dataflow.dflow INFO: Task 972 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbdbb865b36c6f255c451dcb08620472 with result from task 972 parsl.dataflow.dflow DEBUG: Task 972 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 973 has memoization hash 08bc30a0d27fb76a0544a7223a8bac3c parsl.dataflow.memoization INFO: Task 973 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 973 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0e8b0a1f-3fe3-4ede-9866-b2faa225d5c1_characterizeImage_994767_6.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0e8b0a1f-3fe3-4ede-9866-b2faa225d5c1_characterizeImage_994767_6.stderr parsl.dataflow.dflow INFO: Standard output for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/042cac36-6fd3-454d-8e69-7d0bf00b8cc7_isr_994767_6.stdout parsl.dataflow.dflow INFO: Standard error for task 972 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/042cac36-6fd3-454d-8e69-7d0bf00b8cc7_isr_994767_6.stderr parsl.dataflow.dflow INFO: Task 966 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 867555e61398ae10e355e158146a7989 with result from task 966 parsl.dataflow.dflow DEBUG: Task 966 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 967 has memoization hash 37c8081e6cb97e7b93247ebb58cddc00 parsl.dataflow.memoization INFO: Task 967 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 967 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f66e86fa-b6cd-49ab-a5a4-c80153f02934_characterizeImage_994639_43.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f66e86fa-b6cd-49ab-a5a4-c80153f02934_characterizeImage_994639_43.stderr parsl.dataflow.dflow INFO: Standard output for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/9c328e1a-0ff1-4100-a171-bd0bfa5afef5_isr_994639_43.stdout parsl.dataflow.dflow INFO: Standard error for task 966 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/9c328e1a-0ff1-4100-a171-bd0bfa5afef5_isr_994639_43.stderr parsl.dataflow.dflow INFO: Task 978 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f47ded02f2bd184bcb0f58d6b40e3e35 with result from task 978 parsl.dataflow.dflow DEBUG: Task 978 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 979 has memoization hash ada1340060250a49754aa9be57cbfb13 parsl.dataflow.memoization INFO: Task 979 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 979 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/875f4239-c074-4b03-b815-599dd66f8292_characterizeImage_994766_8.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/875f4239-c074-4b03-b815-599dd66f8292_characterizeImage_994766_8.stderr parsl.dataflow.dflow INFO: Standard output for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/99e8a1ee-a647-4ff8-b6cc-4f78246f3757_isr_994766_8.stdout parsl.dataflow.dflow INFO: Standard error for task 978 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/99e8a1ee-a647-4ff8-b6cc-4f78246f3757_isr_994766_8.stderr parsl.dataflow.dflow INFO: Task 969 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7556913dc59a19e8996e7b09c4cac97 with result from task 969 parsl.dataflow.dflow DEBUG: Task 969 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 970 has memoization hash b219695a4c622bf2085d9278cb28c72e parsl.dataflow.memoization INFO: Task 970 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 970 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e5c29989-6b69-4e29-a506-4254454653f9_characterizeImage_994638_46.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e5c29989-6b69-4e29-a506-4254454653f9_characterizeImage_994638_46.stderr parsl.dataflow.dflow INFO: Standard output for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b7d62dd4-875b-428a-80da-405a27bc92af_isr_994638_46.stdout parsl.dataflow.dflow INFO: Standard error for task 969 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b7d62dd4-875b-428a-80da-405a27bc92af_isr_994638_46.stderr parsl.dataflow.dflow INFO: Task 954 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 177ec1df4c5c0cb56ecb2a9b163591db with result from task 954 parsl.dataflow.dflow DEBUG: Task 954 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 955 has memoization hash 83a009936e534668cb7564bbc075ff18 parsl.dataflow.memoization INFO: Task 955 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 955 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8b4cbb2c-78ea-4ec9-b8e8-9f60d88ec480_characterizeImage_994765_21.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8b4cbb2c-78ea-4ec9-b8e8-9f60d88ec480_characterizeImage_994765_21.stderr parsl.dataflow.dflow INFO: Standard output for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fb887205-43b8-400b-b464-bb7269b65e6e_isr_994765_21.stdout parsl.dataflow.dflow INFO: Standard error for task 954 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fb887205-43b8-400b-b464-bb7269b65e6e_isr_994765_21.stderr parsl.dataflow.dflow INFO: Task 960 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 235264a953436d782fb3dcbf3be89ccf with result from task 960 parsl.dataflow.dflow DEBUG: Task 960 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 961 has memoization hash 818a2fa8ed6eced274577f4a27d0e3d0 parsl.dataflow.memoization INFO: Task 961 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 961 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/12b35294-d23b-408e-b524-390548c7ebd2_characterizeImage_994640_38.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/12b35294-d23b-408e-b524-390548c7ebd2_characterizeImage_994640_38.stderr parsl.dataflow.dflow INFO: Standard output for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3de77fe8-0bcd-4ea5-b4fb-f50fc144f9d1_isr_994640_38.stdout parsl.dataflow.dflow INFO: Standard error for task 960 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3de77fe8-0bcd-4ea5-b4fb-f50fc144f9d1_isr_994640_38.stderr parsl.dataflow.dflow INFO: Task 957 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6d7bedca159df4c5beae427bf6c0399 with result from task 957 parsl.dataflow.dflow DEBUG: Task 957 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 958 has memoization hash daca98232a6dbe79952b9ed807725b76 parsl.dataflow.memoization INFO: Task 958 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 958 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9e0f7eb7-0a31-44c8-9247-bb7d17107610_characterizeImage_994638_41.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9e0f7eb7-0a31-44c8-9247-bb7d17107610_characterizeImage_994638_41.stderr parsl.dataflow.dflow INFO: Standard output for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/2d4512ef-f949-4358-aef4-5b023a32ec3c_isr_994638_41.stdout parsl.dataflow.dflow INFO: Standard error for task 957 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/2d4512ef-f949-4358-aef4-5b023a32ec3c_isr_994638_41.stderr parsl.dataflow.dflow INFO: Task 945 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89b59ee006f58b61e0170ef052a1d56d with result from task 945 parsl.dataflow.dflow DEBUG: Task 945 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 946 has memoization hash 3d890dec5221d09e3d11f1f6a2ea3914 parsl.dataflow.memoization INFO: Task 946 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 946 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/678e3e29-3388-49b0-86ef-5fbf1076be5e_characterizeImage_994766_28.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/678e3e29-3388-49b0-86ef-5fbf1076be5e_characterizeImage_994766_28.stderr parsl.dataflow.dflow INFO: Standard output for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/0335de7b-4b43-4459-9be7-955713cebc28_isr_994766_28.stdout parsl.dataflow.dflow INFO: Standard error for task 945 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/0335de7b-4b43-4459-9be7-955713cebc28_isr_994766_28.stderr parsl.dataflow.dflow INFO: Task 951 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06cf90d161523d42ea869c29124a3f9d with result from task 951 parsl.dataflow.dflow DEBUG: Task 951 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 952 has memoization hash e1a96cccfded86bb86d8acd411525587 parsl.dataflow.memoization INFO: Task 952 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 952 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ce7a2248-cb05-4bb8-99e8-9d1a779623a3_characterizeImage_994639_25.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ce7a2248-cb05-4bb8-99e8-9d1a779623a3_characterizeImage_994639_25.stderr parsl.dataflow.dflow INFO: Standard output for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d9795d6d-adeb-4f36-84cb-350d64f1c51b_isr_994639_25.stdout parsl.dataflow.dflow INFO: Standard error for task 951 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/d9795d6d-adeb-4f36-84cb-350d64f1c51b_isr_994639_25.stderr parsl.dataflow.dflow INFO: Task 942 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e0af4e798cd301f287f3e26c14383af with result from task 942 parsl.dataflow.dflow DEBUG: Task 942 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 943 has memoization hash 84b11041ff285128766aac6a6da50791 parsl.dataflow.memoization INFO: Task 943 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 943 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/bcc5439b-e38e-4559-ae8c-2eb23b6c6bab_characterizeImage_994767_3.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/bcc5439b-e38e-4559-ae8c-2eb23b6c6bab_characterizeImage_994767_3.stderr parsl.dataflow.dflow INFO: Standard output for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/3cc04a7e-df09-4d80-886c-2165d3199add_isr_994767_3.stdout parsl.dataflow.dflow INFO: Standard error for task 942 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/3cc04a7e-df09-4d80-886c-2165d3199add_isr_994767_3.stderr parsl.dataflow.dflow INFO: Task 939 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8eb5071789a9d7ff9209e6253e8db77 with result from task 939 parsl.dataflow.dflow DEBUG: Task 939 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 940 has memoization hash b9fe8f6227f73ae5f2d8c2c7cdc2a12b parsl.dataflow.memoization INFO: Task 940 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 940 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b89aea31-b52f-484d-a521-a675c676e63b_characterizeImage_994639_55.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b89aea31-b52f-484d-a521-a675c676e63b_characterizeImage_994639_55.stderr parsl.dataflow.dflow INFO: Standard output for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/491008f5-3895-4314-9356-701808f65637_isr_994639_55.stdout parsl.dataflow.dflow INFO: Standard error for task 939 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/491008f5-3895-4314-9356-701808f65637_isr_994639_55.stderr parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90f282d56c67a98b4277164ef7dbed37 with result from task 582 parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash ea48ae85d1342e4694a5932b5aa01265 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b7340884-dcdf-41e2-bb15-aa1a08d29bcf_characterizeImage_994640_2.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b7340884-dcdf-41e2-bb15-aa1a08d29bcf_characterizeImage_994640_2.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/91231bb5-87a2-4654-aec1-7c870a2fe1d2_isr_994640_2.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/91231bb5-87a2-4654-aec1-7c870a2fe1d2_isr_994640_2.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b134e63576f66892500f18cf161df49 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 17db99cb6ac6666c2eb555e4f539866b parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65040e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b997cb63-1bca-4323-9076-00289b72232c_characterizeImage_994639_2.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b997cb63-1bca-4323-9076-00289b72232c_characterizeImage_994639_2.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/466ecde4-095b-4cd1-95db-9744a7bb5606_isr_994639_2.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/466ecde4-095b-4cd1-95db-9744a7bb5606_isr_994639_2.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de7a883431f019a93f7a819782b7bc8a with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 5fc6e25749a5e3476fb5c49baafe4f3a parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5428fed8-d453-4108-9d88-a7a3fd1f820f_characterizeImage_994640_21.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5428fed8-d453-4108-9d88-a7a3fd1f820f_characterizeImage_994640_21.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5e1459bc-e985-49f9-9df7-0f2d3f79bcbc_isr_994640_21.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/5e1459bc-e985-49f9-9df7-0f2d3f79bcbc_isr_994640_21.stderr parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0732f1fcca36e9f18d3ebe0667c3242e with result from task 726 parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:00 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 931382552e23aa6a4f9d71712e13cd09 parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ec8a1919-cb6a-44d5-93d3-dfdd1b44f592_characterizeImage_994765_59.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ec8a1919-cb6a-44d5-93d3-dfdd1b44f592_characterizeImage_994765_59.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/da9cc97d-91d0-4f78-83f5-0612011a319d_isr_994765_59.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/da9cc97d-91d0-4f78-83f5-0612011a319d_isr_994765_59.stderr parsl.dataflow.dflow INFO: Task 993 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f90b94aa611c3fbdce4cf665b803f43 with result from task 993 parsl.dataflow.dflow DEBUG: Task 993 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 994 has memoization hash 203f6b4f0748590d700e70c02bc88937 parsl.dataflow.memoization INFO: Task 994 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ec400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 994 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/fd41a028-0d17-422c-b567-a2bb9049b620_characterizeImage_994638_51.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/fd41a028-0d17-422c-b567-a2bb9049b620_characterizeImage_994638_51.stderr parsl.dataflow.dflow INFO: Standard output for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8e3b0c04-36df-4f5f-ada7-4dd5019ff437_isr_994638_51.stdout parsl.dataflow.dflow INFO: Standard error for task 993 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8e3b0c04-36df-4f5f-ada7-4dd5019ff437_isr_994638_51.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbfa43bf6114be44fe342b6a4c6dbad5 with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash d6618cfcaeca61589ff73ced20921e3f parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63de0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/78b33abb-4a6e-40f2-8dcb-1a2f51dd91b5_characterizeImage_994638_32.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/78b33abb-4a6e-40f2-8dcb-1a2f51dd91b5_characterizeImage_994638_32.stderr parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d3a03c5e-1b49-4a11-8d16-a963b7aced5d_isr_994638_32.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d3a03c5e-1b49-4a11-8d16-a963b7aced5d_isr_994638_32.stderr parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00b4af5246af24d84e2b28d4e5347369 with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 94f6d9a86799a250417636167c5b75cf parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a80f6182-59d8-4bae-a8c0-7eb9f04dd738_characterizeImage_994638_19.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a80f6182-59d8-4bae-a8c0-7eb9f04dd738_characterizeImage_994638_19.stderr parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/97f06f5c-b65c-45e9-8d9a-c948aa8366e8_isr_994638_19.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/97f06f5c-b65c-45e9-8d9a-c948aa8366e8_isr_994638_19.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c794c2bda0345b0776a3e1b9c7ea54d7 with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash 772c56ad436c9981c305f7eea2559dff parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c329804d-19a9-4c53-8bfa-17ca299a1b36_characterizeImage_994765_55.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c329804d-19a9-4c53-8bfa-17ca299a1b36_characterizeImage_994765_55.stderr parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/050ac7da-cefa-41a5-8c3e-fe49618c194e_isr_994765_55.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/050ac7da-cefa-41a5-8c3e-fe49618c194e_isr_994765_55.stderr parsl.dataflow.dflow INFO: Task 948 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cf412c958e1a2d05204dd5f98604d51 with result from task 948 parsl.dataflow.dflow DEBUG: Task 948 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 949 has memoization hash 14471c8893d7903477b31ffc9999fc95 parsl.dataflow.memoization INFO: Task 949 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 949 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/f18301df-8b67-47c7-93db-7116f6f7f587_characterizeImage_994767_59.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/f18301df-8b67-47c7-93db-7116f6f7f587_characterizeImage_994767_59.stderr parsl.dataflow.dflow INFO: Standard output for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ee1c37b0-657a-459c-ba97-f292f87821e3_isr_994767_59.stdout parsl.dataflow.dflow INFO: Standard error for task 948 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/ee1c37b0-657a-459c-ba97-f292f87821e3_isr_994767_59.stderr parsl.dataflow.dflow INFO: Task 1080 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09d34ff29d28deef792bc57c69789822 with result from task 1080 parsl.dataflow.dflow DEBUG: Task 1080 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1081 has memoization hash 06aa79c7efd8624c89ddf5f6b92ddcc5 parsl.dataflow.memoization INFO: Task 1081 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62efce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1081 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/38d40ac3-a536-41b6-aa4b-e73d79b10ce8_characterizeImage_994766_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/38d40ac3-a536-41b6-aa4b-e73d79b10ce8_characterizeImage_994766_33.stderr parsl.dataflow.dflow INFO: Standard output for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8765558c-1fa9-483b-bd46-29d8f43185bf_isr_994766_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1080 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8765558c-1fa9-483b-bd46-29d8f43185bf_isr_994766_33.stderr parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ed9a8ae02dd208fbaeb4c8a709a4b0f with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 9a76206fe0e26b528af82a188b3a4aae parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6652ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/74ad22a1-4185-4bf7-8e5a-85972e528359_characterizeImage_994766_14.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/74ad22a1-4185-4bf7-8e5a-85972e528359_characterizeImage_994766_14.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/3f65deb9-0d94-4d9a-bdbe-b52c749b5eb3_isr_994766_14.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/3f65deb9-0d94-4d9a-bdbe-b52c749b5eb3_isr_994766_14.stderr parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 864621b5dda14265a3e69dc425e9337d with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash b8ba948116f72caf1a54a509948f8909 parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66514e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/73f4b776-193a-4113-964d-bcb8519cc3e3_characterizeImage_994766_2.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/73f4b776-193a-4113-964d-bcb8519cc3e3_characterizeImage_994766_2.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/7f73d2fe-61bf-4246-b52a-649a93e46109_isr_994766_2.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/7f73d2fe-61bf-4246-b52a-649a93e46109_isr_994766_2.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c9ef99a66d32d23f216ab085dbf8320 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash efe8c9a3609cacee4dc157859902cdc2 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6651800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c66dbeca-c198-48fd-b04a-bf6a48f944cf_characterizeImage_994639_26.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c66dbeca-c198-48fd-b04a-bf6a48f944cf_characterizeImage_994639_26.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6bc82448-70e0-4d55-9575-53d88150608e_isr_994639_26.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6bc82448-70e0-4d55-9575-53d88150608e_isr_994639_26.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad719b6d6d777ba489ef8e8426748a8c with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash df3a8d78ebe301183291835df7d390f7 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6652de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/15e52219-0dff-426e-91e0-090be7aacae0_characterizeImage_994765_54.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/15e52219-0dff-426e-91e0-090be7aacae0_characterizeImage_994765_54.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/f66a2d99-61ec-4d55-8cdd-c20149e9a8b1_isr_994765_54.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/f66a2d99-61ec-4d55-8cdd-c20149e9a8b1_isr_994765_54.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92d03fb421460be5897da3135430f262 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash a67a3fcd28c5e6e367fc1b636caf3aff parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e365f4f4-d6d4-4d3f-ba37-84d0867d9366_characterizeImage_994638_31.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e365f4f4-d6d4-4d3f-ba37-84d0867d9366_characterizeImage_994638_31.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/1ca2c971-4e1d-46ff-9a7f-f0c58a0edc02_isr_994638_31.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/1ca2c971-4e1d-46ff-9a7f-f0c58a0edc02_isr_994638_31.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f0e5e478c49952dd5cb33cdefa465e2 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 64fce95d18f1e3c70a078743ab09ce7b parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/93655875-ba59-4e4e-8314-78767fa9c278_characterizeImage_994767_2.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/93655875-ba59-4e4e-8314-78767fa9c278_characterizeImage_994767_2.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/bb50f2bb-36a2-4c79-9874-e0808e009809_isr_994767_2.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/bb50f2bb-36a2-4c79-9874-e0808e009809_isr_994767_2.stderr parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b63fa889abe85826fd32e6277c929ff with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash 4e49f509d15642667f6b6725608509bc parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/7d4d4010-c886-4303-9e93-dbcaaf876174_characterizeImage_994767_19.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/7d4d4010-c886-4303-9e93-dbcaaf876174_characterizeImage_994767_19.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/efc5b6eb-5481-4d36-8e01-45d739531a0f_isr_994767_19.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/efc5b6eb-5481-4d36-8e01-45d739531a0f_isr_994767_19.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe478731c25f4a6b2b7d68ce56f7cd6d with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash d9b34c941fe5df2d89ef0c05241a9fd0 parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e99e5f49-de59-40ca-a027-fa9843974684_characterizeImage_994766_44.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e99e5f49-de59-40ca-a027-fa9843974684_characterizeImage_994766_44.stderr parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d6e1ef57-09d6-47c8-aa31-f52092837faa_isr_994766_44.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/d6e1ef57-09d6-47c8-aa31-f52092837faa_isr_994766_44.stderr parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05ad338881a73afa3fded65f00bc323f with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash dd5b5cedbc526b246fd1c583a95f67eb parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/dc3cc9b3-6ea1-466f-89c0-b191e61b1368_characterizeImage_994639_23.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/dc3cc9b3-6ea1-466f-89c0-b191e61b1368_characterizeImage_994639_23.stderr parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/12a7de2c-05e8-457c-b9ae-56b146925244_isr_994639_23.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/12a7de2c-05e8-457c-b9ae-56b146925244_isr_994639_23.stderr parsl.dataflow.dflow INFO: Task 1005 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 750c52fc8e5b1b0575cdb07fbf76227d with result from task 1005 parsl.dataflow.dflow DEBUG: Task 1005 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1006 has memoization hash f24b84a96494ba4af83580b35dfec597 parsl.dataflow.memoization INFO: Task 1006 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1006 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/3ad9bd48-3dcc-4a50-a302-1e01ae7b7d11_characterizeImage_994766_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/3ad9bd48-3dcc-4a50-a302-1e01ae7b7d11_characterizeImage_994766_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bfb0dedf-f671-4485-be64-4a50a7da7b00_isr_994766_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1005 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bfb0dedf-f671-4485-be64-4a50a7da7b00_isr_994766_42.stderr parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8490be1805e037ab66ca90f2bd0c792d with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash e9feace7141b3550d662da29d2cf063a parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63df600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2b31f3d6-b0bc-40ed-98fa-d7c94fd38e29_characterizeImage_994766_34.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2b31f3d6-b0bc-40ed-98fa-d7c94fd38e29_characterizeImage_994766_34.stderr parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/df4f2266-4250-4c33-a2a7-9178b8857451_isr_994766_34.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/df4f2266-4250-4c33-a2a7-9178b8857451_isr_994766_34.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fbaaaaf024583b623e75f30bceb37c3 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 06a85004a41ad6a28d24beec75840943 parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66511c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/9c29a172-125a-4be1-bfdb-3a92c5867729_characterizeImage_994639_7.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/9c29a172-125a-4be1-bfdb-3a92c5867729_characterizeImage_994639_7.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ae3e1920-dfb0-42fb-b25f-b4793f95435b_isr_994639_7.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ae3e1920-dfb0-42fb-b25f-b4793f95435b_isr_994639_7.stderr parsl.dataflow.dflow INFO: Task 1002 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3491d5e8cc9ed8633baf515bb95524e6 with result from task 1002 parsl.dataflow.dflow DEBUG: Task 1002 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1003 has memoization hash 97f4a11ec5f1db8ac2ff0b6b2b424a75 parsl.dataflow.memoization INFO: Task 1003 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1003 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c8b6515e-a8b1-446d-a423-b1175ebea43c_characterizeImage_994639_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c8b6515e-a8b1-446d-a423-b1175ebea43c_characterizeImage_994639_13.stderr parsl.dataflow.dflow INFO: Standard output for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/affd8635-f64d-455b-8ff7-a42cb34f1c52_isr_994639_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1002 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/affd8635-f64d-455b-8ff7-a42cb34f1c52_isr_994639_13.stderr parsl.dataflow.dflow INFO: Task 996 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b37c5001e2fa9411e547a2a20108171d with result from task 996 parsl.dataflow.dflow DEBUG: Task 996 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 997 has memoization hash f5d39520420570591316d74fcb4b0831 parsl.dataflow.memoization INFO: Task 997 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 997 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/48647308-0fc5-4d8e-8154-3711de360b88_characterizeImage_994765_43.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/48647308-0fc5-4d8e-8154-3711de360b88_characterizeImage_994765_43.stderr parsl.dataflow.dflow INFO: Standard output for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/00169d10-aadc-4dce-92df-dea1658700e4_isr_994765_43.stdout parsl.dataflow.dflow INFO: Standard error for task 996 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/00169d10-aadc-4dce-92df-dea1658700e4_isr_994765_43.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5941fdf04f06a70b811e4a4064c75259 with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash 209696e1c07c024d1f5df2c70ac5d30c parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e36c20c3-cbf0-4f4d-86f0-aa8088dcc063_characterizeImage_994638_45.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e36c20c3-cbf0-4f4d-86f0-aa8088dcc063_characterizeImage_994638_45.stderr parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b6db45a3-5c31-49de-94bb-30ec36b66f12_isr_994638_45.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b6db45a3-5c31-49de-94bb-30ec36b66f12_isr_994638_45.stderr parsl.dataflow.dflow INFO: Task 999 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24af49924a85098c3715717c8b2da6ed with result from task 999 parsl.dataflow.dflow DEBUG: Task 999 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1000 has memoization hash 5166306907906c0087acdfa77312be5a parsl.dataflow.memoization INFO: Task 1000 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1000 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0ffedbe8-2667-47f0-905c-610fa51e178e_characterizeImage_994640_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0ffedbe8-2667-47f0-905c-610fa51e178e_characterizeImage_994640_10.stderr parsl.dataflow.dflow INFO: Standard output for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/0df11c18-734c-4155-8522-03527ec50ca9_isr_994640_10.stdout parsl.dataflow.dflow INFO: Standard error for task 999 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/0df11c18-734c-4155-8522-03527ec50ca9_isr_994640_10.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dd23e6d2b30dd261c6b185f429a23c7 with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 3bb84e0f112c6249b3809720ffe387d3 parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/5d992288-6fa2-4cd4-9336-7cc4fc887bcc_characterizeImage_994765_24.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/5d992288-6fa2-4cd4-9336-7cc4fc887bcc_characterizeImage_994765_24.stderr parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/af55885e-f6db-40d1-8583-c9fba42f4f75_isr_994765_24.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/af55885e-f6db-40d1-8583-c9fba42f4f75_isr_994765_24.stderr parsl.dataflow.dflow INFO: Task 990 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f78602087c3e2a4852e4af040194ab29 with result from task 990 parsl.dataflow.dflow DEBUG: Task 990 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 991 has memoization hash 29e5e4cb3489d38cc496e9a656c09026 parsl.dataflow.memoization INFO: Task 991 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 991 try 0 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8bf8052b-5d04-4aeb-89ae-9640eebcdf91_characterizeImage_994765_47.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8bf8052b-5d04-4aeb-89ae-9640eebcdf91_characterizeImage_994765_47.stderr parsl.dataflow.dflow INFO: Standard output for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87ca0c1f-94e1-4446-aee8-f00084209cf4_isr_994765_47.stdout parsl.dataflow.dflow INFO: Standard error for task 990 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/87ca0c1f-94e1-4446-aee8-f00084209cf4_isr_994765_47.stderr parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 626dcdeb60787c746b2789c33313a4ac with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash b164c5171e98a1c0e912ee2014044517 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b20c8eb3-d565-4a3d-bf37-06ca2436cec5_characterizeImage_994638_39.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b20c8eb3-d565-4a3d-bf37-06ca2436cec5_characterizeImage_994638_39.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fd900e0a-7fb8-49ab-957c-8869a0aca665_isr_994638_39.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fd900e0a-7fb8-49ab-957c-8869a0aca665_isr_994638_39.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8515e6b31fe0b8d9269e5f386f6bfb83 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash 5708edac1d875c88dc77fdfcb6e7ed53 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6652660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/afe2e65a-72d6-4e48-838d-4ce3ea97c3d6_characterizeImage_994766_61.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/afe2e65a-72d6-4e48-838d-4ce3ea97c3d6_characterizeImage_994766_61.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2374bf25-db36-416c-8d8c-fe8b1fcfcffd_isr_994766_61.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2374bf25-db36-416c-8d8c-fe8b1fcfcffd_isr_994766_61.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81d9e40ee7d9dba3dffc9cf96a7957a7 with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash 308f3b6f161145637d172ee3b7bd9393 parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e4e9a73a-3536-47ca-9518-429430b5b8de_characterizeImage_994766_51.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e4e9a73a-3536-47ca-9518-429430b5b8de_characterizeImage_994766_51.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/17d55cc3-ab57-4e53-8501-f69573fa1903_isr_994766_51.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/17d55cc3-ab57-4e53-8501-f69573fa1903_isr_994766_51.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7013691b4fd40ac215c0987943cb05ba with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash 29af2a519024b3c907f46d0ce6cb1014 parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e1ac32e5-9b56-41b5-a37c-bf41c25e39ba_characterizeImage_994640_57.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e1ac32e5-9b56-41b5-a37c-bf41c25e39ba_characterizeImage_994640_57.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/19b9d3dc-ead4-4049-8fba-68a6fc761138_isr_994640_57.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/19b9d3dc-ead4-4049-8fba-68a6fc761138_isr_994640_57.stderr parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70b5571b7ff954a850938af682035a2a with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash 1c112b4d0164fe9d72f3178972c82826 parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a3029440-f6c6-4cac-98ec-add63024ef96_characterizeImage_994765_33.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a3029440-f6c6-4cac-98ec-add63024ef96_characterizeImage_994765_33.stderr parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0033d07b-63e9-4235-a55a-e45b08bf64b5_isr_994765_33.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0033d07b-63e9-4235-a55a-e45b08bf64b5_isr_994765_33.stderr parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4e1cd1a73f4294062621076538996d9 with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 1a65da4aa230736e2343942adfab358d parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63ddd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2c59e152-5f98-4bc1-bdc8-49bd64408e60_characterizeImage_994767_38.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2c59e152-5f98-4bc1-bdc8-49bd64408e60_characterizeImage_994767_38.stderr parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/4eda5726-9ad4-4789-914e-8ee54ca891c4_isr_994767_38.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/4eda5726-9ad4-4789-914e-8ee54ca891c4_isr_994767_38.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cbdbc891db51309caeb566e3df2fd43 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 259317e8782f4adde6db7f16ff078e8f parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6651bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/756de5d3-d58a-4c98-8af4-2bca448ce13e_characterizeImage_994766_3.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/756de5d3-d58a-4c98-8af4-2bca448ce13e_characterizeImage_994766_3.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8ae0ac5c-aeb2-433c-96b3-b281ec4a794e_isr_994766_3.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/8ae0ac5c-aeb2-433c-96b3-b281ec4a794e_isr_994766_3.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42fa655c376eff2bea82267954a81416 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash 2944044a5dff47535182befde4701538 parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c7c5ceb5-1dd8-46c7-8e9c-ff035b868efd_characterizeImage_994767_41.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c7c5ceb5-1dd8-46c7-8e9c-ff035b868efd_characterizeImage_994767_41.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/4ea3c15a-ff84-418f-84fb-31ca52c5df55_isr_994767_41.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/4ea3c15a-ff84-418f-84fb-31ca52c5df55_isr_994767_41.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bee7c25747d10d77592175910e059829 with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash 5cbc06e403e56462450575b0c3b58144 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/beee8377-dc6d-460c-883f-5fff58d46207_characterizeImage_994765_42.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/beee8377-dc6d-460c-883f-5fff58d46207_characterizeImage_994765_42.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a7e9afde-d6fa-438c-a89d-1d77350c774b_isr_994765_42.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a7e9afde-d6fa-438c-a89d-1d77350c774b_isr_994765_42.stderr parsl.dataflow.dflow INFO: Task 1008 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75e574a34d5930a2f3b698b6ac2e0f21 with result from task 1008 parsl.dataflow.dflow DEBUG: Task 1008 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1009 has memoization hash 6b4c30fc656a931fa63f2462a02c559d parsl.dataflow.memoization INFO: Task 1009 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1009 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/dd612f2f-719d-4cb3-99ca-408fdd1f4d58_characterizeImage_994765_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/dd612f2f-719d-4cb3-99ca-408fdd1f4d58_characterizeImage_994765_12.stderr parsl.dataflow.dflow INFO: Standard output for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/e23b2653-75a0-43c7-ab2a-e6792599d8ef_isr_994765_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1008 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/e23b2653-75a0-43c7-ab2a-e6792599d8ef_isr_994765_12.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7daa2fba2e0d9bf7dafdf62dd989087 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 3efa2c0bd1a7f0b43d6a4007e79b6e3c parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/2d156ecd-7970-4580-a3cd-264de6b6e548_characterizeImage_994638_34.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/2d156ecd-7970-4580-a3cd-264de6b6e548_characterizeImage_994638_34.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/21be9294-61fc-4d9c-b380-9f2df20ec71a_isr_994638_34.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/21be9294-61fc-4d9c-b380-9f2df20ec71a_isr_994638_34.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 850fdd4eb604b344dcdac93322208af9 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash 27aaa1272bcb2d74ae25e1b58e932544 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64847c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2766688a-e143-4ac2-af32-b84b52a76b29_characterizeImage_994640_61.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2766688a-e143-4ac2-af32-b84b52a76b29_characterizeImage_994640_61.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/8907bb10-f2a9-472f-8e0c-2b86fef05248_isr_994640_61.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/8907bb10-f2a9-472f-8e0c-2b86fef05248_isr_994640_61.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1e5836f5292325d3a1c242b05dcc0a7 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 066a65b75e7c080f73d8d2b7dfbe274d parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a833502c-cfae-452f-b312-328db72b955a_characterizeImage_994766_55.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a833502c-cfae-452f-b312-328db72b955a_characterizeImage_994766_55.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bf10c3a4-80f8-423f-a1be-435fdf6cc9e0_isr_994766_55.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bf10c3a4-80f8-423f-a1be-435fdf6cc9e0_isr_994766_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:02 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1011 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6240333f319e974f1a0bec30673c0875 with result from task 1011 parsl.dataflow.dflow DEBUG: Task 1011 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1012 has memoization hash 38423bbcaeeca94a76b40e2a0f6f8a25 parsl.dataflow.memoization INFO: Task 1012 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a72e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1012 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d5d8e699-3acc-4f95-8001-a078b45fbe09_characterizeImage_994639_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d5d8e699-3acc-4f95-8001-a078b45fbe09_characterizeImage_994639_4.stderr parsl.dataflow.dflow INFO: Standard output for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/16c640bd-930b-4ef3-b49e-e6a5daca1fc9_isr_994639_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1011 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/16c640bd-930b-4ef3-b49e-e6a5daca1fc9_isr_994639_4.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cff6588e0d96f4b987bde9b0e0943c6c with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash fc0cdb09d3e835fa9703c0b535c23bd7 parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63df920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b993b482-107b-432e-94d5-b78e70dfdfc8_characterizeImage_994640_48.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b993b482-107b-432e-94d5-b78e70dfdfc8_characterizeImage_994640_48.stderr parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4ff18f0f-5d82-4298-9e45-4b00b635c150_isr_994640_48.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4ff18f0f-5d82-4298-9e45-4b00b635c150_isr_994640_48.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab0cdb658346035566cef26509e03605 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash b0fe93b086769d4f1caf2ff7f5df991a parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c8c1bc09-c19e-4303-933c-d2742446db55_characterizeImage_994640_51.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c8c1bc09-c19e-4303-933c-d2742446db55_characterizeImage_994640_51.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/b31ded7d-c8d8-4c3b-918c-e327d1591a96_isr_994640_51.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/b31ded7d-c8d8-4c3b-918c-e327d1591a96_isr_994640_51.stderr parsl.dataflow.dflow INFO: Task 1104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7315f67355755df3d7ce70759417691 with result from task 1104 parsl.dataflow.dflow DEBUG: Task 1104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 670bdac47e3c118575dcb15a0abb4599 parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9d3d106b-0aca-4ac0-ad90-088f5cd478d7_characterizeImage_994638_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9d3d106b-0aca-4ac0-ad90-088f5cd478d7_characterizeImage_994638_2.stderr parsl.dataflow.dflow INFO: Standard output for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f46c4725-ced0-4461-a6a2-f46ec7401492_isr_994638_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f46c4725-ced0-4461-a6a2-f46ec7401492_isr_994638_2.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8191b65d0f3f4b54d3a8eb8b461f0feb with result from task 771 parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:03 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash d44cddf05022eda1600ad39636c350db parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/be4bd50e-c61f-4fb2-b9db-883bcd01b443_characterizeImage_994638_12.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/be4bd50e-c61f-4fb2-b9db-883bcd01b443_characterizeImage_994638_12.stderr parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ec5e98b5-813b-41c7-9699-5a51299a1773_isr_994638_12.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/ec5e98b5-813b-41c7-9699-5a51299a1773_isr_994638_12.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7788d81533e571c135b35689ed0136f with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 367d37b1d1a213a3d8782c1571835018 parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/41d6c60d-0213-4037-b88f-648bd384721b_characterizeImage_994639_17.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/41d6c60d-0213-4037-b88f-648bd384721b_characterizeImage_994639_17.stderr parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/5358044d-62e1-41de-8086-f43432d4e6f5_isr_994639_17.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/5358044d-62e1-41de-8086-f43432d4e6f5_isr_994639_17.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ad4e46a568b73aac5d15e535a2e1d01 with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash fa5e34874f2a70b714a9eedf13581a01 parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c854eb81-4cf9-4185-9e99-9f519ca8ef43_characterizeImage_994639_38.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c854eb81-4cf9-4185-9e99-9f519ca8ef43_characterizeImage_994639_38.stderr parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/817b0797-232e-4302-b45d-89157a3af5e3_isr_994639_38.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/817b0797-232e-4302-b45d-89157a3af5e3_isr_994639_38.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce1def2c1cbda62368127019ba2386aa with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash b80e16b63a54d4470fff949e6687d20b parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c61016b9-fdfe-4a12-93d0-5aa66d1de94e_characterizeImage_994765_1.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c61016b9-fdfe-4a12-93d0-5aa66d1de94e_characterizeImage_994765_1.stderr parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/02dfd48f-2674-473f-be04-7c1e2b0d0bef_isr_994765_1.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/02dfd48f-2674-473f-be04-7c1e2b0d0bef_isr_994765_1.stderr parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06d29f5b317e52f10ae4d88249927a9f with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash 3520e5cdfd008c702bbc34fef374bbe9 parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/cc14be5f-5733-46cf-9c61-ad8b1c199c01_characterizeImage_994766_41.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/cc14be5f-5733-46cf-9c61-ad8b1c199c01_characterizeImage_994766_41.stderr parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/525b1d2d-cf87-4daf-b0aa-149b006f1866_isr_994766_41.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/525b1d2d-cf87-4daf-b0aa-149b006f1866_isr_994766_41.stderr parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c820aff632fc602ec3774c74f5cde7 with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash 9c4ddbb33b57aed787c86a8619720047 parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f5902395-89fe-4e75-96d5-05806be005d3_characterizeImage_994638_55.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f5902395-89fe-4e75-96d5-05806be005d3_characterizeImage_994638_55.stderr parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8ca8649a-4dce-4b2a-aa2d-3f9ddde65cf1_isr_994638_55.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/8ca8649a-4dce-4b2a-aa2d-3f9ddde65cf1_isr_994638_55.stderr parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de8df5b6b2e61fb5eb4d0100cbe9bcdd with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 4d95ae7f930e3f11a8585963741c4571 parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/242a0e9f-12c4-4563-a445-895b63195a2d_characterizeImage_994638_57.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/242a0e9f-12c4-4563-a445-895b63195a2d_characterizeImage_994638_57.stderr parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/95ee00da-fbf8-4009-8af3-e7a870ea9e00_isr_994638_57.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/95ee00da-fbf8-4009-8af3-e7a870ea9e00_isr_994638_57.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45a7d0191598984214eb095c6b98e14f with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash 0de569e57e416ef2fe760d7d646af4c0 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f7ba5002-edfc-48a5-b758-3efdb6d2d305_characterizeImage_994638_23.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f7ba5002-edfc-48a5-b758-3efdb6d2d305_characterizeImage_994638_23.stderr parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fc16a0c9-27cd-4186-8dab-391bd90151e0_isr_994638_23.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/fc16a0c9-27cd-4186-8dab-391bd90151e0_isr_994638_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:03 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfd6b22f2ea2d3a8bc6e10f1064820b0 with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash ae1ad0fe244425d7b4fefcf8c1d7b712 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/3af779ef-2a16-4814-ae73-42a33c0bea62_characterizeImage_994766_38.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/3af779ef-2a16-4814-ae73-42a33c0bea62_characterizeImage_994766_38.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/7420c4d5-a3a4-4732-a433-01770c58e23d_isr_994766_38.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/7420c4d5-a3a4-4732-a433-01770c58e23d_isr_994766_38.stderr parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ed728379c0c74435e8e8d1979e77cda with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 38813d25f52f821ac8054b2a942dd487 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/1e87736f-7b05-4232-af58-757531e61f7e_characterizeImage_994640_18.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/1e87736f-7b05-4232-af58-757531e61f7e_characterizeImage_994640_18.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3a8f288f-64bb-467f-9a77-aa420357a090_isr_994640_18.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/3a8f288f-64bb-467f-9a77-aa420357a090_isr_994640_18.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bf4350142b1471879592f8e2ab512ce with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash dfc9b03fe1f39ae1ea15b752d9c07d51 parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f06b531a-211f-4be4-94fd-3f16d16bb34a_characterizeImage_994766_54.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f06b531a-211f-4be4-94fd-3f16d16bb34a_characterizeImage_994766_54.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b873dd31-7f91-42e4-bc4a-2c70346cbd48_isr_994766_54.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b873dd31-7f91-42e4-bc4a-2c70346cbd48_isr_994766_54.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae8b02c788a8ccb1a454c0e52d6be460 with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 39ee442de2b9643f48aeaddc1d603edc parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/882e3ca2-460b-43bf-80c6-2393f27ffa69_characterizeImage_994638_11.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/882e3ca2-460b-43bf-80c6-2393f27ffa69_characterizeImage_994638_11.stderr parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/cff76e4b-4b64-4be5-812d-2d25079d2ea9_isr_994638_11.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/cff76e4b-4b64-4be5-812d-2d25079d2ea9_isr_994638_11.stderr parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a468127a98120be2cc68ef0846d4334 with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 1d5450e139d6edc046b57c76776ece76 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/63017076-4c6a-4833-a6e8-975967ff6587_characterizeImage_994638_49.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/63017076-4c6a-4833-a6e8-975967ff6587_characterizeImage_994638_49.stderr parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/e16a2c8a-5318-4656-88f5-b851a0b0831f_isr_994638_49.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/e16a2c8a-5318-4656-88f5-b851a0b0831f_isr_994638_49.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1846dbeb6e0b64496e6d5b0bf860b536 with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash 2a6a9c27a755349e8c374466421eea1f parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63de3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d510ddeb-3dae-4e43-8df3-be2383f7c5ff_characterizeImage_994639_10.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d510ddeb-3dae-4e43-8df3-be2383f7c5ff_characterizeImage_994639_10.stderr parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6e877c52-ceeb-4d89-91d3-b8fe0a2611b6_isr_994639_10.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6e877c52-ceeb-4d89-91d3-b8fe0a2611b6_isr_994639_10.stderr parsl.dataflow.dflow INFO: Task 1077 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 101a8dafa70af3794f713f035c337187 with result from task 1077 parsl.dataflow.dflow DEBUG: Task 1077 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1078 has memoization hash 2af9e8306e22d11405d119a55beb3099 parsl.dataflow.memoization INFO: Task 1078 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ef9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1078 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/8f268223-a8a8-4f6f-92d5-e5fc77164969_characterizeImage_994638_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/8f268223-a8a8-4f6f-92d5-e5fc77164969_characterizeImage_994638_62.stderr parsl.dataflow.dflow INFO: Standard output for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/51895857-607c-40a1-981c-896f655b5a9e_isr_994638_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1077 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/51895857-607c-40a1-981c-896f655b5a9e_isr_994638_62.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1857b15b375b72f1723fcbc00705e221 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash 5405e6edba159e7b865817f83e8a5e4e parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0d449e69-98ca-4522-a95f-679fc9f80c17_characterizeImage_994767_1.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0d449e69-98ca-4522-a95f-679fc9f80c17_characterizeImage_994767_1.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d4319505-1367-463d-8341-a060c9f4f27f_isr_994767_1.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d4319505-1367-463d-8341-a060c9f4f27f_isr_994767_1.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1ec88679ce84544ccb071b80b18a99c with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 99090aea498c3cb8cc6595e4152eb4c7 parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/dda9ace1-c631-4efd-a366-220b46db6d2f_characterizeImage_994765_35.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/dda9ace1-c631-4efd-a366-220b46db6d2f_characterizeImage_994765_35.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fcf79596-189b-48f1-984b-f76e0a92ec04_isr_994765_35.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fcf79596-189b-48f1-984b-f76e0a92ec04_isr_994765_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:04 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc3bfcb7cbd22f9bf70351ce897d6519 with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 6f13e34850c22282a215a9ebcdb62f3e parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/20ebe835-f66e-41c4-8a6e-9c9f3fc5e836_characterizeImage_994640_54.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/20ebe835-f66e-41c4-8a6e-9c9f3fc5e836_characterizeImage_994640_54.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e2230cf2-038e-46eb-80ea-51b0758eec42_isr_994640_54.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/e2230cf2-038e-46eb-80ea-51b0758eec42_isr_994640_54.stderr parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2af7935273c33699aad47ca8adb15fe4 with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash 64b23e5a868e741df5c3a627d77c05cd parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/7b4214db-bbf2-4796-952a-7ba81e272228_characterizeImage_994766_6.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/7b4214db-bbf2-4796-952a-7ba81e272228_characterizeImage_994766_6.stderr parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b5670708-1919-4fa3-8a2b-49354f8918bd_isr_994766_6.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b5670708-1919-4fa3-8a2b-49354f8918bd_isr_994766_6.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95be6f089a66c69977df0aa034d01b10 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash b89dfe2375a21cfdac98e9c830239746 parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0fea87bc-ce23-41d4-acbc-6d11002decf6_characterizeImage_994639_40.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0fea87bc-ce23-41d4-acbc-6d11002decf6_characterizeImage_994639_40.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6709cc75-74f0-4e1e-a6f1-3e08ce9c12d3_isr_994639_40.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6709cc75-74f0-4e1e-a6f1-3e08ce9c12d3_isr_994639_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:04 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71c34819c0f2c809155401b0e2bdf6fe with result from task 1107 parsl.dataflow.dflow DEBUG: Task 1107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1108 has memoization hash a4e2b69dfe4040112ebdd4b0ac8f0728 parsl.dataflow.memoization INFO: Task 1108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1108 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/62b0291d-ca93-43d9-a8a5-173c8f238cfd_characterizeImage_994767_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/62b0291d-ca93-43d9-a8a5-173c8f238cfd_characterizeImage_994767_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/be35aa9d-706b-4395-b092-c159c9b0eb0b_isr_994767_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/be35aa9d-706b-4395-b092-c159c9b0eb0b_isr_994767_45.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6d8ef900bc2b01cfc635684c340bf22 with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash c4c42d15234ca5f2f5452ecf77e0d57c parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c16a81cb-c0e8-460d-90b2-c2fd6107ed3b_characterizeImage_994640_28.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c16a81cb-c0e8-460d-90b2-c2fd6107ed3b_characterizeImage_994640_28.stderr parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c912e725-15e2-410f-ad42-64c64aa65d4c_isr_994640_28.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/c912e725-15e2-410f-ad42-64c64aa65d4c_isr_994640_28.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d86e08b70babaf7f6295a1b464f6a492 with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash 72f5f729a9f3af8215b0182308c33273 parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/617c516d-c7e2-46a6-bf1b-d32b42b96e33_characterizeImage_994766_1.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/617c516d-c7e2-46a6-bf1b-d32b42b96e33_characterizeImage_994766_1.stderr parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b11a8cde-8ac5-4af3-ba48-9705d9061e8c_isr_994766_1.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/b11a8cde-8ac5-4af3-ba48-9705d9061e8c_isr_994766_1.stderr parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64f1e25f024e3f9723950d9e32fc594e with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash 921a474913d852ea5ca2f0343ef9fbc0 parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63937e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/92a455c5-25dc-4289-9eb6-bc932840321f_characterizeImage_994640_56.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/92a455c5-25dc-4289-9eb6-bc932840321f_characterizeImage_994640_56.stderr parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f4996f73-618c-4dd3-9832-d5238b62ff1f_isr_994640_56.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f4996f73-618c-4dd3-9832-d5238b62ff1f_isr_994640_56.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f8e499fa565d5e9152b87060e522619 with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash e0e0b91e2525e41c8e19e53ca1d0a50c parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4893b070-e191-4c12-943f-d7e36dc75dd9_characterizeImage_994767_52.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4893b070-e191-4c12-943f-d7e36dc75dd9_characterizeImage_994767_52.stderr parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8648817b-1f52-4add-85a1-905dd5a4af48_isr_994767_52.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/8648817b-1f52-4add-85a1-905dd5a4af48_isr_994767_52.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79527440cf094cf167014b5cd2ac20e6 with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash b0b2d412baa2319915cd95faa71cb998 parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63ddb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a0ee579b-7bde-407b-993f-9cb0cb3f084e_characterizeImage_994638_43.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a0ee579b-7bde-407b-993f-9cb0cb3f084e_characterizeImage_994638_43.stderr parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b92997b7-e554-4200-b33e-67422c668ddb_isr_994638_43.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/b92997b7-e554-4200-b33e-67422c668ddb_isr_994638_43.stderr parsl.dataflow.dflow INFO: Task 1110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0b9b6ddd6c9b7ff6a611fc1f1a3a111 with result from task 1110 parsl.dataflow.dflow DEBUG: Task 1110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1111 has memoization hash 6e56bde084c498cf1bb15e5bd415413b parsl.dataflow.memoization INFO: Task 1111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6346020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1111 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d38bfc30-235c-4f6d-9d78-74a033186470_characterizeImage_994765_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d38bfc30-235c-4f6d-9d78-74a033186470_characterizeImage_994765_32.stderr parsl.dataflow.dflow INFO: Standard output for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/74a2414e-3798-4cee-a342-2c11463ff863_isr_994765_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/74a2414e-3798-4cee-a342-2c11463ff863_isr_994765_32.stderr parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5e0feb5a8f00273a4081783492bc927 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash d43c875c912cd19b0a0816ebe0b1d41e parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64be200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2ba2f722-bd0d-46ad-bbbe-e6cf87667d97_characterizeImage_994766_48.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2ba2f722-bd0d-46ad-bbbe-e6cf87667d97_characterizeImage_994766_48.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/19dd2d23-7d9c-461e-ba24-1a8dc92a2b9b_isr_994766_48.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/19dd2d23-7d9c-461e-ba24-1a8dc92a2b9b_isr_994766_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:05 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f710fe6b4d2d09ce45e62f986c667f61 with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash 4582a64f3d725731119698b5c9e0ebe7 parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/112a3be1-4aeb-4e7e-a48a-e6ce62ee40b5_characterizeImage_994766_9.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/112a3be1-4aeb-4e7e-a48a-e6ce62ee40b5_characterizeImage_994766_9.stderr parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bd828462-8a77-4638-8ac6-f05a4c859409_isr_994766_9.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/bd828462-8a77-4638-8ac6-f05a4c859409_isr_994766_9.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5b076cf8400f62123ee90a028ffdedd with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 412189598951d684dfd68960263bd0f0 parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/0abeab3f-d2ec-40b2-9f40-02eecc84c771_characterizeImage_994765_9.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/0abeab3f-d2ec-40b2-9f40-02eecc84c771_characterizeImage_994765_9.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cb41f1b5-e7a2-47b3-a714-2b4f060f55e0_isr_994765_9.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cb41f1b5-e7a2-47b3-a714-2b4f060f55e0_isr_994765_9.stderr parsl.dataflow.dflow INFO: Task 1092 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13fc4f53f2e6f21f555c57b4e17b8212 with result from task 1092 parsl.dataflow.dflow DEBUG: Task 1092 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1093 has memoization hash 2baba51f2d5d99f623b2244630d50539 parsl.dataflow.memoization INFO: Task 1093 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63445e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1093 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/54a2fbba-c726-4338-adaf-442383716b15_characterizeImage_994639_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/54a2fbba-c726-4338-adaf-442383716b15_characterizeImage_994639_49.stderr parsl.dataflow.dflow INFO: Standard output for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/19f91076-1ae6-4605-a728-bbf140d2d79e_isr_994639_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1092 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/19f91076-1ae6-4605-a728-bbf140d2d79e_isr_994639_49.stderr parsl.dataflow.dflow INFO: Task 1071 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa9c6f2823edd5483eaac59bf51f72c5 with result from task 1071 parsl.dataflow.dflow DEBUG: Task 1071 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1072 has memoization hash 9cbf151198e90665bdc22ad47bf4c626 parsl.dataflow.memoization INFO: Task 1072 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ef2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1072 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4c223ea1-2512-4acc-b339-95f56eb07dfa_characterizeImage_994767_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4c223ea1-2512-4acc-b339-95f56eb07dfa_characterizeImage_994767_26.stderr parsl.dataflow.dflow INFO: Standard output for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/3a028f41-1582-4640-8b95-d69a60e12da7_isr_994767_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1071 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/3a028f41-1582-4640-8b95-d69a60e12da7_isr_994767_26.stderr parsl.dataflow.dflow INFO: Task 1074 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0843b366231cd1b78c043e90314ad672 with result from task 1074 parsl.dataflow.dflow DEBUG: Task 1074 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1075 has memoization hash 8a4e19f7f149df86c163c494bc52566d parsl.dataflow.memoization INFO: Task 1075 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ef6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1075 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/bffd2e48-52e4-404f-a803-736405ef1b5a_characterizeImage_994638_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/bffd2e48-52e4-404f-a803-736405ef1b5a_characterizeImage_994638_38.stderr parsl.dataflow.dflow INFO: Standard output for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/964e4059-084d-488b-a8bc-6e7a4e3d433c_isr_994638_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1074 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/964e4059-084d-488b-a8bc-6e7a4e3d433c_isr_994638_38.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dbe282ad302ea9b7a2503485d9b3ea4 with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash b2fd20f9d225e29e07d4777399ba2f22 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6603b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/d56796d4-a339-4428-94ef-507b069a3bbd_characterizeImage_994638_53.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/d56796d4-a339-4428-94ef-507b069a3bbd_characterizeImage_994638_53.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/955fc71c-86e6-4e18-8d88-6070a65b41dc_isr_994638_53.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/955fc71c-86e6-4e18-8d88-6070a65b41dc_isr_994638_53.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42850edf15390d8f0842dcf48c438757 with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash 4d975d1d2c27aeede90a2ec9b1093dac parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63de700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e6ed663b-11f6-4971-a214-53bafda1b55a_characterizeImage_994638_52.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e6ed663b-11f6-4971-a214-53bafda1b55a_characterizeImage_994638_52.stderr parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6a21db39-074b-4956-8a4c-6efbc5fc534b_isr_994638_52.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/6a21db39-074b-4956-8a4c-6efbc5fc534b_isr_994638_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:05 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e45b0034cf8c3380303309e4849de02a with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash c80f7dbfeeddde315a6f93c22c8af13e parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63def20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8546076c-eee2-49c9-9942-09b0f4ddc680_characterizeImage_994767_7.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8546076c-eee2-49c9-9942-09b0f4ddc680_characterizeImage_994767_7.stderr parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/dde15828-2f8b-4998-afe2-2abd56fde916_isr_994767_7.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/dde15828-2f8b-4998-afe2-2abd56fde916_isr_994767_7.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 890d601ed29a00d7296c41835cc97350 with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 5c81ad662f45998bb48a26da75e95b5f parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f224ba4b-334c-421e-83aa-f2dafd507a08_characterizeImage_994765_11.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f224ba4b-334c-421e-83aa-f2dafd507a08_characterizeImage_994765_11.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cc0a214f-c6ed-44ff-8cec-4eda2dbada76_isr_994765_11.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/cc0a214f-c6ed-44ff-8cec-4eda2dbada76_isr_994765_11.stderr parsl.dataflow.dflow INFO: Task 1089 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95eea96c099d43f66cb8572e60a14b53 with result from task 1089 parsl.dataflow.dflow DEBUG: Task 1089 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1090 has memoization hash 87d456b582a27753dcffef3c1411e288 parsl.dataflow.memoization INFO: Task 1090 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63442c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1090 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/56411ec4-2621-4f17-a373-cecce63eec39_characterizeImage_994640_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/56411ec4-2621-4f17-a373-cecce63eec39_characterizeImage_994640_12.stderr parsl.dataflow.dflow INFO: Standard output for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/87c0c29f-27fe-48a7-a2ae-a28b87d93271_isr_994640_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1089 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/87c0c29f-27fe-48a7-a2ae-a28b87d93271_isr_994640_12.stderr parsl.dataflow.dflow INFO: Task 1083 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94ebb1094c70eecf36c10dbb57ff5a9c with result from task 1083 parsl.dataflow.dflow DEBUG: Task 1083 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1084 has memoization hash 01c82e7391d5197b1db3bd1af05ebfc4 parsl.dataflow.memoization INFO: Task 1084 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62eff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1084 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/95de70d5-4289-4d30-968a-e81e640d9371_characterizeImage_994640_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/95de70d5-4289-4d30-968a-e81e640d9371_characterizeImage_994640_26.stderr parsl.dataflow.dflow INFO: Standard output for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/693a14ae-b533-417e-b99b-40a137489bc0_isr_994640_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1083 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/693a14ae-b533-417e-b99b-40a137489bc0_isr_994640_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:06 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fada77e476be206cec41e2ca692509d with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash f8e0dfe93fed216ed3b87f003b5a4d4e parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/66437cfd-f083-43f6-8540-4fba32ea09b6_characterizeImage_994640_55.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/66437cfd-f083-43f6-8540-4fba32ea09b6_characterizeImage_994640_55.stderr parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4c8c5ccd-91a0-4979-98d2-5b48103c6225_isr_994640_55.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4c8c5ccd-91a0-4979-98d2-5b48103c6225_isr_994640_55.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9ed65c8fc0a31564266003de6689554 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 2d270cdb3a7cf28ef494a35a02726610 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66031a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ed5a4ea9-6a70-4f07-8921-3618d4ae288d_characterizeImage_994639_27.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ed5a4ea9-6a70-4f07-8921-3618d4ae288d_characterizeImage_994639_27.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/f48399ba-e7d1-4e81-9973-f11f935f5665_isr_994639_27.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/f48399ba-e7d1-4e81-9973-f11f935f5665_isr_994639_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:06 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f7f3bfe0f0d586365cf352887154983 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 166a77cbec5d8b955782f97d4ca84780 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66037e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8f10ce28-85a0-46d7-aafe-f989c3089c47_characterizeImage_994639_34.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8f10ce28-85a0-46d7-aafe-f989c3089c47_characterizeImage_994639_34.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/1095c75f-9b25-407a-bbb7-3ecaeddd66e0_isr_994639_34.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/1095c75f-9b25-407a-bbb7-3ecaeddd66e0_isr_994639_34.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81c3e26e3a3c36feb0fe1d94f749d861 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash f906e1acffe620a756c8747a41ce59de parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6603e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c31afe7c-4b76-47e1-b5cc-1ec7cf091952_characterizeImage_994767_39.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c31afe7c-4b76-47e1-b5cc-1ec7cf091952_characterizeImage_994767_39.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5d96ff8b-9a82-4591-8a4d-ef428c7d2e23_isr_994767_39.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5d96ff8b-9a82-4591-8a4d-ef428c7d2e23_isr_994767_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:07 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1086 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a95567619ff7291b6b68743da4fab74d with result from task 1086 parsl.dataflow.dflow DEBUG: Task 1086 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1087 has memoization hash bcb8a177f1c95ffddbd2f50b8c90b4c9 parsl.dataflow.memoization INFO: Task 1087 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1087 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/10de8e7b-2a10-46f0-8872-0099a1cd51d2_characterizeImage_994638_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/10de8e7b-2a10-46f0-8872-0099a1cd51d2_characterizeImage_994638_58.stderr parsl.dataflow.dflow INFO: Standard output for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d4a7ff74-c5d8-4cd3-bfa6-4befcfb368fd_isr_994638_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1086 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/d4a7ff74-c5d8-4cd3-bfa6-4befcfb368fd_isr_994638_58.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abb2c4d63a09d09e769eda19972ede7d with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 3ef077c54381c6fab3893b24a1f415b3 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/847de146-552e-41c0-a879-16150e744761_characterizeImage_994767_49.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/847de146-552e-41c0-a879-16150e744761_characterizeImage_994767_49.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/68902548-c7e9-432c-8098-e859658c8156_isr_994767_49.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/68902548-c7e9-432c-8098-e859658c8156_isr_994767_49.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 148386701b5beba803f0386d7589cbdc with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 49e7dfcd367a1f754e04b01e9719fce3 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ff16fbcd-e446-4b1f-a1d4-60dfe78232ad_characterizeImage_994639_32.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ff16fbcd-e446-4b1f-a1d4-60dfe78232ad_characterizeImage_994639_32.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ce1b7ea9-bdff-41b8-b936-5d4b60df1cdd_isr_994639_32.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/ce1b7ea9-bdff-41b8-b936-5d4b60df1cdd_isr_994639_32.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 379fbf225477aa06dd0d960fb69f3697 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash b5ee838b48e02787e41ad586bb061524 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2b9dd577-a70f-449d-ac96-98abf4b1ffbb_characterizeImage_994640_13.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2b9dd577-a70f-449d-ac96-98abf4b1ffbb_characterizeImage_994640_13.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f29a70e6-b19b-4f06-9441-b5637175a26e_isr_994640_13.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/f29a70e6-b19b-4f06-9441-b5637175a26e_isr_994640_13.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 702bb5d069086b8c8b277565c0e84027 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 852d30804cd59dcfea0aea824223118f parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66034c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/77ee0189-fde6-4dc1-aaca-a5f9558207fe_characterizeImage_994766_57.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/77ee0189-fde6-4dc1-aaca-a5f9558207fe_characterizeImage_994766_57.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/692f36c8-c0a6-434e-8006-c29bd1e99829_isr_994766_57.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/692f36c8-c0a6-434e-8006-c29bd1e99829_isr_994766_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Task 1047 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97757d58dcefee3acdc77d44d0ea2dc6 with result from task 1047 parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 372 parsl.dataflow.dflow DEBUG: Task 1047 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1048 has memoization hash 48da6e15b8e00add84841a09094fbd15 parsl.dataflow.memoization INFO: Task 1048 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ede40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1048 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/e7646def-0cea-4061-b622-78d92b6fe98b_characterizeImage_994765_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/e7646def-0cea-4061-b622-78d92b6fe98b_characterizeImage_994765_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/dd02bcd2-298e-4294-abc6-562f2d026ac3_isr_994765_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1047 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/dd02bcd2-298e-4294-abc6-562f2d026ac3_isr_994765_56.stderr parsl.dataflow.strategy DEBUG: Executor multi has 372 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:14:07 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b93ba7038945bb6c1e5ef6965aa9b0f with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash 91eea19ce8d980b87d19fe068737d37e parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64be8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/39126770-b77d-4fb1-a9c6-3a07a8fd68a0_characterizeImage_994638_40.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/39126770-b77d-4fb1-a9c6-3a07a8fd68a0_characterizeImage_994638_40.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/31ac6abc-9a5f-443a-aec3-0be283e4b2a3_isr_994638_40.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/31ac6abc-9a5f-443a-aec3-0be283e4b2a3_isr_994638_40.stderr parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 478be3ada8194b2b63b30ca5ca2626c3 with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash 336e7c76f8d0af33dad1da0faf8ac3f5 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a14105f6-206f-4666-918a-08d07d96b699_characterizeImage_994766_15.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a14105f6-206f-4666-918a-08d07d96b699_characterizeImage_994766_15.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/30329358-91c6-4f50-a61f-2eca1c0881e2_isr_994766_15.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/30329358-91c6-4f50-a61f-2eca1c0881e2_isr_994766_15.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae74c21818097139e2d86bc75d4a0984 with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 356a5c04a9f67b7594e94cc6f222066d parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3c1a8bc2-d505-4cb0-a024-4ddff488798e_characterizeImage_994639_48.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3c1a8bc2-d505-4cb0-a024-4ddff488798e_characterizeImage_994639_48.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6c7d34e5-170e-41f0-8e2c-1030fe415b4b_isr_994639_48.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/6c7d34e5-170e-41f0-8e2c-1030fe415b4b_isr_994639_48.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de7445a2b08d87ac7046ec0a940b8661 with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash 7b68f5d7e8c2f60bfd30883997d69c45 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/5094fd64-88b9-4d8c-8adc-e39a8b843616_characterizeImage_994638_5.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/5094fd64-88b9-4d8c-8adc-e39a8b843616_characterizeImage_994638_5.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/62d2426d-a441-4e05-b2ef-43e58ba15514_isr_994638_5.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/62d2426d-a441-4e05-b2ef-43e58ba15514_isr_994638_5.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8df9875dd7e0e5fd9200c6a799be5c5c with result from task 702 2024-03-15 04:14:07 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash 249eff07d0e7468b9964b0fd979a59ba parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/58a0c1ba-43b8-4444-88b5-26b426a971b7_characterizeImage_994766_5.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/58a0c1ba-43b8-4444-88b5-26b426a971b7_characterizeImage_994766_5.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2c20c85f-e5d1-45f7-9b93-5db1aba4158e_isr_994766_5.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/2c20c85f-e5d1-45f7-9b93-5db1aba4158e_isr_994766_5.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94fac503517bc2a5873199ffd717ac70 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 2ced15700c178eacea6a9a60745a0cc2 parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64be520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/00cf9354-9dd6-429d-9713-e0ed6b2a8702_characterizeImage_994767_35.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/00cf9354-9dd6-429d-9713-e0ed6b2a8702_characterizeImage_994767_35.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/e1c87191-863a-4936-9845-3807a6d75aa9_isr_994767_35.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/e1c87191-863a-4936-9845-3807a6d75aa9_isr_994767_35.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b65827959fa99f126dd34a86926456e with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash c5757d694219fb189a2663a9d3cd0a7f parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/33d6aaa2-49b3-45a8-a904-8db2d5dfcece_characterizeImage_994765_14.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/33d6aaa2-49b3-45a8-a904-8db2d5dfcece_characterizeImage_994765_14.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0ee59462-217f-4f28-a0f6-2c86b85b5b2a_isr_994765_14.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0ee59462-217f-4f28-a0f6-2c86b85b5b2a_isr_994765_14.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6b7aa5fa00b4858060938e6b56c2ee5 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash c9a5b53d04f58b9036973fb6d5c72c21 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/750e181e-3b75-4420-bb25-eefe43be964a_characterizeImage_994639_8.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/750e181e-3b75-4420-bb25-eefe43be964a_characterizeImage_994639_8.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/4d340158-6fb8-4be3-a06b-d63f39e35d3e_isr_994639_8.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/4d340158-6fb8-4be3-a06b-d63f39e35d3e_isr_994639_8.stderr parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a371beda1e3b89f9a0e2cbc3bd34e873 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 8641190fe0c3076b85d0ccb79ac3b3fc parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f165efcf-3589-4119-b9f7-66a3696e704b_characterizeImage_994638_47.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f165efcf-3589-4119-b9f7-66a3696e704b_characterizeImage_994638_47.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/897bf95c-12d7-4b74-ae15-8b8f981fe8a3_isr_994638_47.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/897bf95c-12d7-4b74-ae15-8b8f981fe8a3_isr_994638_47.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1688898cfae3474919250098fca4749 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash ed92dc7422db111c5de891cbbc2bf381 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/43a765cf-cb37-41a9-8a8c-55209d01851c_characterizeImage_994640_14.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/43a765cf-cb37-41a9-8a8c-55209d01851c_characterizeImage_994640_14.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/56092868-1d27-45bf-9546-3681595eacbc_isr_994640_14.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/56092868-1d27-45bf-9546-3681595eacbc_isr_994640_14.stderr parsl.dataflow.dflow INFO: Task 1062 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d68c14b589d61a6a3621440cdc232808 with result from task 1062 parsl.dataflow.dflow DEBUG: Task 1062 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1063 has memoization hash eec7a2b833b1c7872f78fc43993d7d22 parsl.dataflow.memoization INFO: Task 1063 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1063 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ea5c66d0-930b-4646-841e-93cf99dcb49a_characterizeImage_994638_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ea5c66d0-930b-4646-841e-93cf99dcb49a_characterizeImage_994638_35.stderr parsl.dataflow.dflow INFO: Standard output for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/84a0b3fb-898d-43ca-bb02-8b8a0c3a693c_isr_994638_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1062 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/84a0b3fb-898d-43ca-bb02-8b8a0c3a693c_isr_994638_35.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fd41111dab3202ec857897c56823a30 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 70d716c3aab54bf141a34f6e30b3b41f parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a3c3865f-59b6-4afc-bb76-376839ac3d08_characterizeImage_994765_4.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a3c3865f-59b6-4afc-bb76-376839ac3d08_characterizeImage_994765_4.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/af870c7d-b995-47d3-b123-7f761c23bda0_isr_994765_4.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/af870c7d-b995-47d3-b123-7f761c23bda0_isr_994765_4.stderr parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff11aab5189e7c6fa46d5d5227ca7704 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 03c8b96af1f50e97421172faac315c94 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/66b39563-e945-416a-93e5-1345eeccb909_characterizeImage_994765_16.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/66b39563-e945-416a-93e5-1345eeccb909_characterizeImage_994765_16.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/4d65e774-3d13-495c-bbab-b34a52de1ffb_isr_994765_16.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/4d65e774-3d13-495c-bbab-b34a52de1ffb_isr_994765_16.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3567bb08747b2ac3d11ed47427de77d with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash 3b301f85cabfb12fb6e670c62724edd8 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/d4f06e18-e9aa-4239-9a3d-4bcff67c53a8_characterizeImage_994640_16.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/d4f06e18-e9aa-4239-9a3d-4bcff67c53a8_characterizeImage_994640_16.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/bae3303a-66a2-45cc-b78b-b677b3547eb7_isr_994640_16.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/bae3303a-66a2-45cc-b78b-b677b3547eb7_isr_994640_16.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32c2b4630dffefdb87738b59453eb952 with result from task 564 parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:08 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash ee7034d15509c666ac7567c3044d339c parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/fecbaa73-e12d-4c85-921c-5c356a0308b9_characterizeImage_994640_45.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/fecbaa73-e12d-4c85-921c-5c356a0308b9_characterizeImage_994640_45.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/32544663-ce9d-4489-95d5-7139c45e1821_isr_994640_45.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/32544663-ce9d-4489-95d5-7139c45e1821_isr_994640_45.stderr parsl.dataflow.dflow INFO: Task 1044 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 774e18f253794470e1ed2f971eae161f with result from task 1044 parsl.dataflow.dflow DEBUG: Task 1044 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1045 has memoization hash d5906f3436b1901075f2ed4995b4e202 parsl.dataflow.memoization INFO: Task 1045 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62edb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1045 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/01c8d3dd-d764-4573-b638-c9745c38969e_characterizeImage_994765_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/01c8d3dd-d764-4573-b638-c9745c38969e_characterizeImage_994765_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fc94ce19-3dc1-4501-8b82-2e8bfe7f4f53_isr_994765_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1044 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/fc94ce19-3dc1-4501-8b82-2e8bfe7f4f53_isr_994765_30.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 627ba287687e1edd0854b430d18edaa3 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 0be005014c0386b0a5b6f6eafd5b1070 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6603f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/a1d65427-2805-4157-852f-e270dd882009_characterizeImage_994640_23.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/a1d65427-2805-4157-852f-e270dd882009_characterizeImage_994640_23.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d82c0c64-6a0b-47a6-84c1-6cc647448d61_isr_994640_23.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/d82c0c64-6a0b-47a6-84c1-6cc647448d61_isr_994640_23.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7fa29459c4ed1c3eb7c9fd734b8e7f6 with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 17f89a395181cd4b83ec59cd8241a495 parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/1065cc2a-c864-4fb1-b3af-b8ec1aa6dd5b_characterizeImage_994640_49.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/1065cc2a-c864-4fb1-b3af-b8ec1aa6dd5b_characterizeImage_994640_49.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/2c30a0dc-2b00-4615-a4a7-6eead8027c9c_isr_994640_49.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/2c30a0dc-2b00-4615-a4a7-6eead8027c9c_isr_994640_49.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22d3f89f462163a263b54885aead3335 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 7f0cd7f90d515eacde99243767912379 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/67a3f952-9dad-4ccc-b0a2-1f46e10b41b5_characterizeImage_994639_62.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/67a3f952-9dad-4ccc-b0a2-1f46e10b41b5_characterizeImage_994639_62.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/2f95c209-cb14-40f5-9731-1bfbaa96eaed_isr_994639_62.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/2f95c209-cb14-40f5-9731-1bfbaa96eaed_isr_994639_62.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c31f129ed6bc4e079439241e1e50f2c0 with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 2e977c61dfb118c4ddae251aeaf690fd parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5e03f5da-0c50-4734-bfca-3cf6f3deca50_characterizeImage_994639_60.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5e03f5da-0c50-4734-bfca-3cf6f3deca50_characterizeImage_994639_60.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/38b12774-aef3-4efa-9b05-b02694beb311_isr_994639_60.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/38b12774-aef3-4efa-9b05-b02694beb311_isr_994639_60.stderr parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a660376fff4b56e507acaab9ccb8c311 with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash 850ce0b8959438d370759aa2ed1637a0 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f0a179c7-5b2e-4fe2-a7ca-f7bc0c1f7a9d_characterizeImage_994766_17.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f0a179c7-5b2e-4fe2-a7ca-f7bc0c1f7a9d_characterizeImage_994766_17.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/be831f52-feb1-49bd-8d1c-0442cf909755_isr_994766_17.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/be831f52-feb1-49bd-8d1c-0442cf909755_isr_994766_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:09 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49659ec53717ff42b1f45943ee926e6f with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash 87105170aec62fd2ec3c77bdf35c805a parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/4995ebc0-1b0c-4771-8a05-1c1f20ecb7e3_characterizeImage_994640_1.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/4995ebc0-1b0c-4771-8a05-1c1f20ecb7e3_characterizeImage_994640_1.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/705bd872-743c-4447-a793-f5f927beb0a9_isr_994640_1.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/705bd872-743c-4447-a793-f5f927beb0a9_isr_994640_1.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ea88d802b68fb4fa3aac7d055ebd66b with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash 656fd9639960b0aaa883f22d80f57c1e parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ea1eed1a-eae2-4107-965a-632e7affa3fa_characterizeImage_994640_29.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ea1eed1a-eae2-4107-965a-632e7affa3fa_characterizeImage_994640_29.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/cbf5b206-0469-4ae2-97fa-5460d1fc3e26_isr_994640_29.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/cbf5b206-0469-4ae2-97fa-5460d1fc3e26_isr_994640_29.stderr parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b548b7f685d3d4d464d796444bf1f93 with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash 61d93563a8dcb030be9b334511e68b7d parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/67e0afd5-d295-4762-8823-3dec2f32cc34_characterizeImage_994766_12.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/67e0afd5-d295-4762-8823-3dec2f32cc34_characterizeImage_994766_12.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4347b490-3351-4905-a902-3b7393d1cd2c_isr_994766_12.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/4347b490-3351-4905-a902-3b7393d1cd2c_isr_994766_12.stderr parsl.dataflow.dflow INFO: Task 1053 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eee5629d5f927b69ac4b5becc833d166 with result from task 1053 parsl.dataflow.dflow DEBUG: Task 1053 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1054 has memoization hash 0d472406b7f3f5b6831404850fc0b697 parsl.dataflow.memoization INFO: Task 1054 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ee480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1054 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/dcf33530-7751-46d9-b4ae-a1d4e0c84738_characterizeImage_994638_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/dcf33530-7751-46d9-b4ae-a1d4e0c84738_characterizeImage_994638_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f83f46e5-6a9f-4d86-a754-981911549b31_isr_994638_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1053 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/f83f46e5-6a9f-4d86-a754-981911549b31_isr_994638_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:09 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1059 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f2caa49b4658f0ee71f950aaf585b99 with result from task 1059 parsl.dataflow.dflow DEBUG: Task 1059 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1060 has memoization hash 77975428e412b727eb8be987ccfd04a0 parsl.dataflow.memoization INFO: Task 1060 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62eeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1060 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8439cb46-0a01-4611-a40a-4ecacd70d3fd_characterizeImage_994767_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8439cb46-0a01-4611-a40a-4ecacd70d3fd_characterizeImage_994767_12.stderr parsl.dataflow.dflow INFO: Standard output for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/98f33088-cfd9-401f-831d-c9cd4342ae90_isr_994767_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1059 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/98f33088-cfd9-401f-831d-c9cd4342ae90_isr_994767_12.stderr parsl.dataflow.dflow INFO: Task 1056 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a00cfd8f1ed912959c497f24883d461 with result from task 1056 parsl.dataflow.dflow DEBUG: Task 1056 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1057 has memoization hash 55dd99cc5c70d3c44dadab49d25014d0 parsl.dataflow.memoization INFO: Task 1057 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ee7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1057 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/193590eb-490a-423c-827a-b4f57d1be6e6_characterizeImage_994639_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/193590eb-490a-423c-827a-b4f57d1be6e6_characterizeImage_994639_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/96e57bd6-cf09-4eb5-b368-8420306649d5_isr_994639_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1056 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/96e57bd6-cf09-4eb5-b368-8420306649d5_isr_994639_42.stderr parsl.dataflow.dflow INFO: Task 1050 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3c5d5c929b39cb97ee492ec23edd1fe with result from task 1050 parsl.dataflow.dflow DEBUG: Task 1050 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1051 has memoization hash 44ac161be677512574f62ca22d3f4573 parsl.dataflow.memoization INFO: Task 1051 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ee160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1051 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5a493726-ba9b-4f66-ac60-e3fc283ee052_characterizeImage_994639_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5a493726-ba9b-4f66-ac60-e3fc283ee052_characterizeImage_994639_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/3f6beff2-9718-49f6-84fc-883386ea47dc_isr_994639_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1050 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/3f6beff2-9718-49f6-84fc-883386ea47dc_isr_994639_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:10 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:10 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea08ba7b322dc9be0dd67ba5734db6d1 with result from task 1113 parsl.dataflow.dflow DEBUG: Task 1113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1114 has memoization hash fc4d43ec41c31144e50a0b083eccf0f7 parsl.dataflow.memoization INFO: Task 1114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6346340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1114 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5a2ac9b2-36cf-4815-aa5a-91b01a773eea_characterizeImage_994640_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5a2ac9b2-36cf-4815-aa5a-91b01a773eea_characterizeImage_994640_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4531eb06-ab8f-4e4e-9e1c-9167ea9c2573_isr_994640_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/4531eb06-ab8f-4e4e-9e1c-9167ea9c2573_isr_994640_47.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 76 try 1 failed parsl.dataflow.dflow ERROR: Task 76 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0c480b101e15696aa976ad188dcde24e with result from task 76 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 77 try 0 failed parsl.dataflow.dflow INFO: Task 77 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/d97d580c-f2b7-43dc-b539-cc78a128056b_calibrate_994765_2.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/d97d580c-f2b7-43dc-b539-cc78a128056b_calibrate_994765_2.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a2701b9f-845a-43f3-bafd-8d03975dedc0_characterizeImage_994765_2.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a2701b9f-845a-43f3-bafd-8d03975dedc0_characterizeImage_994765_2.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4546a1a9b3b94455bf85171de000ecd3 with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash c624f13885eff196402bbb5044271756 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:11 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/dcfaf225-f135-4b67-85f3-ed6cfcab55d8_characterizeImage_994767_58.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/dcfaf225-f135-4b67-85f3-ed6cfcab55d8_characterizeImage_994767_58.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/0dd62aea-726c-414c-ba53-27383b90aae9_isr_994767_58.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/0dd62aea-726c-414c-ba53-27383b90aae9_isr_994767_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:11 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1065 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b0768d26dd8be298677211ef8574479 with result from task 1065 parsl.dataflow.dflow DEBUG: Task 1065 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1066 has memoization hash 603cd55301d06c642886429aca449444 parsl.dataflow.memoization INFO: Task 1066 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1066 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/d3225329-7277-481c-8fc7-b689d542403e_characterizeImage_994638_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/d3225329-7277-481c-8fc7-b689d542403e_characterizeImage_994638_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/30a9316c-0065-428e-b49a-a1a8e0bd2333_isr_994638_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1065 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/30a9316c-0065-428e-b49a-a1a8e0bd2333_isr_994638_37.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 499 try 0 failed parsl.dataflow.dflow INFO: Task 499 marked for retry parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/93655875-ba59-4e4e-8314-78767fa9c278_characterizeImage_994767_2.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/93655875-ba59-4e4e-8314-78767fa9c278_characterizeImage_994767_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash 64fce95d18f1e3c70a078743ab09ce7b parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 1 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/93655875-ba59-4e4e-8314-78767fa9c278_characterizeImage_994767_2.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/93655875-ba59-4e4e-8314-78767fa9c278_characterizeImage_994767_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:12 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 439 try 0 failed parsl.dataflow.dflow INFO: Task 439 marked for retry parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/73f4b776-193a-4113-964d-bcb8519cc3e3_characterizeImage_994766_2.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/73f4b776-193a-4113-964d-bcb8519cc3e3_characterizeImage_994766_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash b8ba948116f72caf1a54a509948f8909 parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66514e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 1 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/73f4b776-193a-4113-964d-bcb8519cc3e3_characterizeImage_994766_2.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/73f4b776-193a-4113-964d-bcb8519cc3e3_characterizeImage_994766_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:12 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:13 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:13 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcce452a3d8c6e8c48ad6ef6959ea498 with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 0ca1d490157eccf7cd24963a7acf4b6c parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62696c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c1444872-1699-4752-bb0f-0526b4478caf_characterizeImage_994767_33.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c1444872-1699-4752-bb0f-0526b4478caf_characterizeImage_994767_33.stderr parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5db7f05d-f601-4ba1-900f-da3403b851ba_isr_994767_33.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5db7f05d-f601-4ba1-900f-da3403b851ba_isr_994767_33.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bdce1c2a0ca7bbdf32b879dea0f9e75 with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash f442f2f93a16460907e9f6c28e19d631 parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62687c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/3b63b58e-c890-4a91-8625-5ecac3923d9d_characterizeImage_994640_19.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/3b63b58e-c890-4a91-8625-5ecac3923d9d_characterizeImage_994640_19.stderr parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/603939b0-3596-4516-a143-8694abe075cb_isr_994640_19.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/603939b0-3596-4516-a143-8694abe075cb_isr_994640_19.stderr parsl.dataflow.dflow INFO: Task 936 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e527c4a11413f003f1a2db4863303e26 with result from task 936 parsl.dataflow.dflow DEBUG: Task 936 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 937 has memoization hash fbefec9a7a22f0894f580b02fc8dab66 parsl.dataflow.memoization INFO: Task 937 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 937 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/26947abe-eb5f-4921-a26e-c09421121f23_characterizeImage_994638_18.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/26947abe-eb5f-4921-a26e-c09421121f23_characterizeImage_994638_18.stderr parsl.dataflow.dflow INFO: Standard output for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/cd04f394-037e-4369-947b-e461d1f5db6c_isr_994638_18.stdout parsl.dataflow.dflow INFO: Standard error for task 936 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/cd04f394-037e-4369-947b-e461d1f5db6c_isr_994638_18.stderr parsl.dataflow.dflow INFO: Task 930 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f3f87f3a8ee02f83664be6fbd7e0d78 with result from task 930 parsl.dataflow.dflow DEBUG: Task 930 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 931 has memoization hash 8cafbd8d59aff5c3878c8d1b18d50a64 parsl.dataflow.memoization INFO: Task 931 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 931 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/966d854a-e15d-46b3-b75a-297272c4309a_characterizeImage_994765_46.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/966d854a-e15d-46b3-b75a-297272c4309a_characterizeImage_994765_46.stderr parsl.dataflow.dflow INFO: Standard output for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9586f860-c742-479f-af80-fdd4b665f9be_isr_994765_46.stdout parsl.dataflow.dflow INFO: Standard error for task 930 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9586f860-c742-479f-af80-fdd4b665f9be_isr_994765_46.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5797cd8d5f3549cd720abcdc4937c42 with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 1baf5e00dfeb50ec06b8ef9105e4064d parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/bfa2d793-4073-403b-b8f2-29e4408b3b24_characterizeImage_994640_5.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/bfa2d793-4073-403b-b8f2-29e4408b3b24_characterizeImage_994640_5.stderr parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/332cacf8-9326-4724-ab64-4160c0cd0d52_isr_994640_5.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/332cacf8-9326-4724-ab64-4160c0cd0d52_isr_994640_5.stderr parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcbe0c44fd8b82a736987b14d94e8aaf with result from task 894 parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash 4576f01870643b04220cc8a53f788603 parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b674087d-4466-4bd1-82ca-d267bcc9f71e_characterizeImage_994639_30.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b674087d-4466-4bd1-82ca-d267bcc9f71e_characterizeImage_994639_30.stderr parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/3fdf753c-fd33-4166-91af-ae3d8804f338_isr_994639_30.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/3fdf753c-fd33-4166-91af-ae3d8804f338_isr_994639_30.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9da51af4838194d77fd6753dfe279eaa with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash 10c6fa3bc7fa469b6f5b8837716544d9 parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/1a760cff-0631-4fdd-a769-a1e3245df1bb_characterizeImage_994767_13.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/1a760cff-0631-4fdd-a769-a1e3245df1bb_characterizeImage_994767_13.stderr parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/00b2bff6-2833-4464-a5a2-0de045e38394_isr_994767_13.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/00b2bff6-2833-4464-a5a2-0de045e38394_isr_994767_13.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 175150868c7758d8a722d40375f7402b with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash ba370bfbbced7800b8c431438df19971 parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/ddebb8fd-4a56-4a22-83ce-13b29785affc_characterizeImage_994767_15.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/ddebb8fd-4a56-4a22-83ce-13b29785affc_characterizeImage_994767_15.stderr parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9c889813-5068-4d7c-813e-5f6c75608271_isr_994767_15.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/9c889813-5068-4d7c-813e-5f6c75608271_isr_994767_15.stderr parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 065bc8dc150acb9329daad91ba5e8d17 with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 232b3e71d1b16859fc7ba6321f042008 parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/9f5031b1-fff7-4a08-ad6e-2cb92ed76d7a_characterizeImage_994765_15.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/9f5031b1-fff7-4a08-ad6e-2cb92ed76d7a_characterizeImage_994765_15.stderr parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/94919847-6f47-4664-bc76-dbff1649a336_isr_994765_15.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/94919847-6f47-4664-bc76-dbff1649a336_isr_994765_15.stderr parsl.dataflow.dflow INFO: Task 933 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfbb1b2db08446f230d5c6217a9bac73 with result from task 933 parsl.dataflow.dflow DEBUG: Task 933 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 934 has memoization hash 45c5916a0f7bb0c88cddb5fe908636ec parsl.dataflow.memoization INFO: Task 934 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 934 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/159161a6-c36a-48f1-ab15-3390ceb4efe3_characterizeImage_994765_49.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/159161a6-c36a-48f1-ab15-3390ceb4efe3_characterizeImage_994765_49.stderr parsl.dataflow.dflow INFO: Standard output for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/4d35f123-2774-4b0a-ba03-87650cb1967e_isr_994765_49.stdout parsl.dataflow.dflow INFO: Standard error for task 933 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/4d35f123-2774-4b0a-ba03-87650cb1967e_isr_994765_49.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0f0e5123967aa1949a6a8079288ebda with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 2e17fad86c7908f2f3436418f1a99dde parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/b194cd70-ddd2-44c0-ae78-e8029303331c_characterizeImage_994767_50.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/b194cd70-ddd2-44c0-ae78-e8029303331c_characterizeImage_994767_50.stderr parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5f1e4065-4a7e-466d-8c7b-9899729eb344_isr_994767_50.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/5f1e4065-4a7e-466d-8c7b-9899729eb344_isr_994767_50.stderr parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10a47b71a8ee2b5044520dfc850959bc with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash 4d57ec6a8e30f06ce81dbbdf797f058c parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/2be6d322-6f97-4d02-a91a-8f587a219b64_characterizeImage_994765_58.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/2be6d322-6f97-4d02-a91a-8f587a219b64_characterizeImage_994765_58.stderr parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0209fa9a-3438-405a-835b-fdc0646bbbc0_isr_994765_58.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/0209fa9a-3438-405a-835b-fdc0646bbbc0_isr_994765_58.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 595 try 0 failed parsl.dataflow.dflow INFO: Task 595 marked for retry parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b997cb63-1bca-4323-9076-00289b72232c_characterizeImage_994639_2.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b997cb63-1bca-4323-9076-00289b72232c_characterizeImage_994639_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 17db99cb6ac6666c2eb555e4f539866b parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65040e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 1 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b997cb63-1bca-4323-9076-00289b72232c_characterizeImage_994639_2.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b997cb63-1bca-4323-9076-00289b72232c_characterizeImage_994639_2.stderr parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d080e10f15dd48492225a27d70623574 with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 69c6b71f07d58f4f52e4a74715282f8a parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/43bfaf91-afe4-4db0-8e8f-5ffeab3e3cc7_characterizeImage_994640_42.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/43bfaf91-afe4-4db0-8e8f-5ffeab3e3cc7_characterizeImage_994640_42.stderr parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/08aeec8a-8509-4590-b1c8-3effe2d0da3b_isr_994640_42.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/08aeec8a-8509-4590-b1c8-3effe2d0da3b_isr_994640_42.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 583 try 0 failed parsl.dataflow.dflow INFO: Task 583 marked for retry parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b7340884-dcdf-41e2-bb15-aa1a08d29bcf_characterizeImage_994640_2.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b7340884-dcdf-41e2-bb15-aa1a08d29bcf_characterizeImage_994640_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash ea48ae85d1342e4694a5932b5aa01265 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 1 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b7340884-dcdf-41e2-bb15-aa1a08d29bcf_characterizeImage_994640_2.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b7340884-dcdf-41e2-bb15-aa1a08d29bcf_characterizeImage_994640_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:14 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3e060a24981d6b84929bbc0587353c6 with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash 228d32d0a1d9f0aee1b942e46b160100 parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/29f21edc-a976-486e-8681-7df49faff034_characterizeImage_994640_60.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/29f21edc-a976-486e-8681-7df49faff034_characterizeImage_994640_60.stderr parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/47b87710-95f3-4e73-a8a9-e850bc9f32f4_isr_994640_60.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994640/47b87710-95f3-4e73-a8a9-e850bc9f32f4_isr_994640_60.stderr parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 327265bb71d3e6ff605b6f68a76dc36c with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash c8e17535c4bc1f6a88e37aaa40eca3a6 parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/e9dfa96b-a3f6-4380-a314-2948cda9dc18_characterizeImage_994765_36.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/e9dfa96b-a3f6-4380-a314-2948cda9dc18_characterizeImage_994765_36.stderr parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/dad5dece-4e19-41dc-b7e9-baf57dbe17a9_isr_994765_36.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/dad5dece-4e19-41dc-b7e9-baf57dbe17a9_isr_994765_36.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1105 try 0 failed parsl.dataflow.dflow INFO: Task 1105 marked for retry parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9d3d106b-0aca-4ac0-ad90-088f5cd478d7_characterizeImage_994638_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9d3d106b-0aca-4ac0-ad90-088f5cd478d7_characterizeImage_994638_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1105 has memoization hash 670bdac47e3c118575dcb15a0abb4599 parsl.dataflow.memoization INFO: Task 1105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1105 try 1 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9d3d106b-0aca-4ac0-ad90-088f5cd478d7_characterizeImage_994638_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9d3d106b-0aca-4ac0-ad90-088f5cd478d7_characterizeImage_994638_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:14 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6a463fa6e8d70960697cc5245209eb8 with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash 2a328eb052cb498a03d539467c78dc6d parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/739013a6-4dd3-4ec9-babc-e68a32441261_characterizeImage_994638_44.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/739013a6-4dd3-4ec9-babc-e68a32441261_characterizeImage_994638_44.stderr parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3fa92cd0-91b5-453d-bde0-6287b4644cbd_isr_994638_44.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994638/3fa92cd0-91b5-453d-bde0-6287b4644cbd_isr_994638_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:15 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:15 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:16 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:16 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:17 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:17 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:18 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:18 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:19 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:20 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:20 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1032 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05c353f282724f9f2c1ff00fe51ebf51 with result from task 1032 parsl.dataflow.dflow DEBUG: Task 1032 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1033 has memoization hash 96c89343f6cc11e3758d94140c612351 parsl.dataflow.memoization INFO: Task 1033 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ecea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1033 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/b5591803-ca16-479b-82ed-b3cbdc87b315_characterizeImage_994767_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/b5591803-ca16-479b-82ed-b3cbdc87b315_characterizeImage_994767_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d10b7fa8-b697-4556-924c-e96bad362ca9_isr_994767_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1032 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994767/d10b7fa8-b697-4556-924c-e96bad362ca9_isr_994767_22.stderr parsl.dataflow.dflow INFO: Task 1041 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cbe903c4ca27c8c26a81a5fb0b25da3 with result from task 1041 parsl.dataflow.dflow DEBUG: Task 1041 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1042 has memoization hash e061f084006a1a8b9798e40b18cf54f5 parsl.dataflow.memoization INFO: Task 1042 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ed800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1042 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f563be75-e0d3-47fe-9a96-a09f2c9b4210_characterizeImage_994765_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f563be75-e0d3-47fe-9a96-a09f2c9b4210_characterizeImage_994765_13.stderr parsl.dataflow.dflow INFO: Standard output for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b69a1df8-aed9-4714-a750-7e2768f93ec8_isr_994765_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1041 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/b69a1df8-aed9-4714-a750-7e2768f93ec8_isr_994765_13.stderr parsl.dataflow.dflow INFO: Task 1020 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 114278151bd8453f44b92dc16b379024 with result from task 1020 parsl.dataflow.dflow DEBUG: Task 1020 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1021 has memoization hash 2671b47ba9a052840431c9154f8f6cef parsl.dataflow.memoization INFO: Task 1021 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1021 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/399442c0-995f-4f80-890e-3383d6c0ea8d_characterizeImage_994765_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/399442c0-995f-4f80-890e-3383d6c0ea8d_characterizeImage_994765_17.stderr parsl.dataflow.dflow INFO: Standard output for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9fd801dd-ad2f-4327-aced-0f2d80bbce1a_isr_994765_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1020 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/9fd801dd-ad2f-4327-aced-0f2d80bbce1a_isr_994765_17.stderr parsl.dataflow.dflow INFO: Task 1026 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9dc9aa8376e0d44dc910ab3ad8c413a with result from task 1026 parsl.dataflow.dflow DEBUG: Task 1026 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1027 has memoization hash 9e75357728cb20e8f3e835ef006edff0 parsl.dataflow.memoization INFO: Task 1027 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ec860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1027 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b74f82b3-e449-431c-a08e-f4e9949e363d_characterizeImage_994765_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b74f82b3-e449-431c-a08e-f4e9949e363d_characterizeImage_994765_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/3dd89a7b-d5f7-4df4-bcc3-621b7a0337d0_isr_994765_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1026 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/3dd89a7b-d5f7-4df4-bcc3-621b7a0337d0_isr_994765_8.stderr parsl.dataflow.dflow INFO: Task 1035 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 418e886236a51fbf47cc1dd600390cbb with result from task 1035 parsl.dataflow.dflow DEBUG: Task 1035 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1036 has memoization hash 035f1d88ad9fde31e3f54ae7baa452e5 parsl.dataflow.memoization INFO: Task 1036 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ed1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1036 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/12d5f9d7-57da-491d-a783-06fab4bd441a_characterizeImage_994639_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/12d5f9d7-57da-491d-a783-06fab4bd441a_characterizeImage_994639_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/dc7efac0-0ff2-4881-9637-bace858b5b50_isr_994639_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1035 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994639/dc7efac0-0ff2-4881-9637-bace858b5b50_isr_994639_19.stderr parsl.dataflow.dflow INFO: Task 1029 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f991c10b5b5e51053bff342cce6b9a0 with result from task 1029 parsl.dataflow.dflow DEBUG: Task 1029 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1030 has memoization hash 8c1c1011a108ebd128c91fde30a0b086 parsl.dataflow.memoization INFO: Task 1030 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ecb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1030 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/abc325be-94d4-4d13-8abe-d8ab413cd10d_characterizeImage_994766_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/abc325be-94d4-4d13-8abe-d8ab413cd10d_characterizeImage_994766_46.stderr parsl.dataflow.dflow INFO: Standard output for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/0d8129f6-6696-48b0-b43b-9c42bec4c201_isr_994766_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1029 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/0d8129f6-6696-48b0-b43b-9c42bec4c201_isr_994766_46.stderr parsl.dataflow.dflow INFO: Task 1038 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a75a4414c11e8d0c98ab04d50e72b6d8 with result from task 1038 parsl.dataflow.dflow DEBUG: Task 1038 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1039 has memoization hash 9827304be4177fbe7d3e7819d323c13f parsl.dataflow.memoization INFO: Task 1039 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ed4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1039 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ed7a8b1b-5157-4f6c-92f8-b1dda9bf3ff7_characterizeImage_994765_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ed7a8b1b-5157-4f6c-92f8-b1dda9bf3ff7_characterizeImage_994765_29.stderr parsl.dataflow.dflow INFO: Standard output for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a0c40475-cac8-4838-aa97-8265cde4ddaa_isr_994765_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1038 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994765/a0c40475-cac8-4838-aa97-8265cde4ddaa_isr_994765_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:21 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:21 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1023 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a62f5cf76033efc3da2738086d7331e7 with result from task 1023 parsl.dataflow.dflow DEBUG: Task 1023 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1024 has memoization hash cd541f1733090186ecf03bd53f7597b8 parsl.dataflow.memoization INFO: Task 1024 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1024 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/91af87b8-b39a-4672-9321-582034b92b93_characterizeImage_994766_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/91af87b8-b39a-4672-9321-582034b92b93_characterizeImage_994766_4.stderr parsl.dataflow.dflow INFO: Standard output for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ebd8f983-5a76-4eb5-b660-7d762ac18197_isr_994766_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1023 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/isr/994766/ebd8f983-5a76-4eb5-b660-7d762ac18197_isr_994766_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:22 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:22 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 439 try 1 failed parsl.dataflow.dflow ERROR: Task 439 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry b8ba948116f72caf1a54a509948f8909 with result from task 439 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 440 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 440 try 0 failed parsl.dataflow.dflow INFO: Task 440 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/68e7a221-5062-4955-aa51-367005024110_calibrate_994766_2.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/68e7a221-5062-4955-aa51-367005024110_calibrate_994766_2.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/73f4b776-193a-4113-964d-bcb8519cc3e3_characterizeImage_994766_2.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/73f4b776-193a-4113-964d-bcb8519cc3e3_characterizeImage_994766_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:23 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:23 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:24 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:24 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:25 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:25 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:26 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:26 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:27 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:28 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:29 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 112baadff15761f3482fd62702ac7b7d with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 258f73ce40d6d9de80135eb5daa818bf parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8a7f2cd2-7464-426e-a76b-0f285a0abdd0_calibrate_994767_51.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8a7f2cd2-7464-426e-a76b-0f285a0abdd0_calibrate_994767_51.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c2bba4cb-c1b8-46fa-a23b-c4cedbe44421_characterizeImage_994767_51.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c2bba4cb-c1b8-46fa-a23b-c4cedbe44421_characterizeImage_994767_51.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e249de90cd6e4aecce6c65b260821799 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 3d35eda497a977d298632c110b4b220a parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c9d00fcf-5d7e-4c29-ae87-d878f19cb5c0_calibrate_994765_60.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c9d00fcf-5d7e-4c29-ae87-d878f19cb5c0_calibrate_994765_60.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/7595d8e4-146b-4179-ba51-6b61377157bd_characterizeImage_994765_60.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/7595d8e4-146b-4179-ba51-6b61377157bd_characterizeImage_994765_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:29 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59c643ab7fb52346a0c7ddd4489a1f8d with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 3412b380d31dd54b0bd62d09cfbe8577 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65063e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4bca1ca4-35cd-41c5-a5fe-1e117217d095_calibrate_994765_51.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4bca1ca4-35cd-41c5-a5fe-1e117217d095_calibrate_994765_51.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/0220d844-dc44-4c17-946f-9432c00bbb02_characterizeImage_994765_51.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/0220d844-dc44-4c17-946f-9432c00bbb02_characterizeImage_994765_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:30 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:30 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1524161761b4e566e9e8af8462968960 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash c5950b8110d6f76b5dc820ff4f145a9a parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a6d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7ea29db2-f0c7-4d73-8814-4ab8a5cce799_calibrate_994767_25.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7ea29db2-f0c7-4d73-8814-4ab8a5cce799_calibrate_994767_25.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/97c7ceef-235c-4076-b0eb-b3b120a9c706_characterizeImage_994767_25.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/97c7ceef-235c-4076-b0eb-b3b120a9c706_characterizeImage_994767_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:31 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e8ce30f6bd93573fb5b63b60e3fe59e with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 7989dc47a63ac8961a74d8ea23b833f5 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2ef91ce7-6b77-4412-9b63-9f09b65109ad_calibrate_994767_62.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2ef91ce7-6b77-4412-9b63-9f09b65109ad_calibrate_994767_62.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/75518ea5-d557-478d-bf11-1af26df7ac88_characterizeImage_994767_62.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/75518ea5-d557-478d-bf11-1af26df7ac88_characterizeImage_994767_62.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d6bd59fabfc3a126f2036a52812d4ab with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 2b49386fb5d6a911655911a15e038105 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ea324901-ae37-4a76-8052-370a3aaa93ed_calibrate_994765_62.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ea324901-ae37-4a76-8052-370a3aaa93ed_calibrate_994765_62.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/95456d32-0010-4291-8d7a-7933366226ff_characterizeImage_994765_62.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/95456d32-0010-4291-8d7a-7933366226ff_characterizeImage_994765_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:31 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7ea77dd2751cd142c193660d74b4eac with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash cf8ade217ba9f8040a211893ec3a8f57 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ce06c7e0-2d6a-4174-8331-a2f8dfc26f8c_calibrate_994766_21.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ce06c7e0-2d6a-4174-8331-a2f8dfc26f8c_calibrate_994766_21.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f9f72975-c9f5-47b2-9362-688945ddb623_characterizeImage_994766_21.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f9f72975-c9f5-47b2-9362-688945ddb623_characterizeImage_994766_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:32 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 499 try 1 failed parsl.dataflow.dflow ERROR: Task 499 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 64fce95d18f1e3c70a078743ab09ce7b with result from task 499 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 500 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 500 try 0 failed parsl.dataflow.dflow INFO: Task 500 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/4c984183-cf8a-42de-ba17-a4cabb2d9662_calibrate_994767_2.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/4c984183-cf8a-42de-ba17-a4cabb2d9662_calibrate_994767_2.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/93655875-ba59-4e4e-8314-78767fa9c278_characterizeImage_994767_2.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/93655875-ba59-4e4e-8314-78767fa9c278_characterizeImage_994767_2.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 794a669a240ef6d7aa6df1b92da30e43 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 593b6a9dc0e55da76699840b0936cc30 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/77c0460b-409e-4ff5-9849-0dbab3f1f680_calibrate_994766_10.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/77c0460b-409e-4ff5-9849-0dbab3f1f680_calibrate_994766_10.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5593bc1a-e52a-49f5-b59d-f339d9bc4efe_characterizeImage_994766_10.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5593bc1a-e52a-49f5-b59d-f339d9bc4efe_characterizeImage_994766_10.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 049e00e6dba93d66f53618929d980345 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash cb48e2b3af5dde3b35913883d7a62fc4 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6e2d0e9b-d004-47cd-96c4-5e9ca894f387_calibrate_994766_26.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6e2d0e9b-d004-47cd-96c4-5e9ca894f387_calibrate_994766_26.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/54f55163-541e-4af0-98a2-9aace25cf3cd_characterizeImage_994766_26.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/54f55163-541e-4af0-98a2-9aace25cf3cd_characterizeImage_994766_26.stderr parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edba5b927df503d246a31e59dbcdc161 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 20b3a05066d3eff7851abdd8b1a2b167 parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f8bbc962-f5e0-434b-b268-5423a7ea6e20_calibrate_994765_10.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f8bbc962-f5e0-434b-b268-5423a7ea6e20_calibrate_994765_10.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/faa468f8-1d91-496b-908d-135642a00b58_characterizeImage_994765_10.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/faa468f8-1d91-496b-908d-135642a00b58_characterizeImage_994765_10.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcf146fba7299e82262728da368ed89a with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 16cc4e4d839a36eda348fe3ec1584365 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66016c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1229f817-55d4-435c-8b0b-3e20c5fbf081_calibrate_994767_60.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1229f817-55d4-435c-8b0b-3e20c5fbf081_calibrate_994767_60.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0faf4069-2af1-4f46-9843-7243775fed0a_characterizeImage_994767_60.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0faf4069-2af1-4f46-9843-7243775fed0a_characterizeImage_994767_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:34 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1434d3b3a2f69ae4886aef112c0abc1 with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash 81673456172120bda496382d437596b4 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/400873f3-88d4-4206-93e8-fef23c91fe5f_calibrate_994765_40.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/400873f3-88d4-4206-93e8-fef23c91fe5f_calibrate_994765_40.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/6026d1e9-c50c-4b5b-9c83-79f1fd1bd6e8_characterizeImage_994765_40.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/6026d1e9-c50c-4b5b-9c83-79f1fd1bd6e8_characterizeImage_994765_40.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f173a4bd1a6bf85c1b8a7f3868ebdff0 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash a8ef07bdf86a390797152fd095c53e38 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4a468632-ae10-48f7-ab12-06aafca0130d_calibrate_994765_57.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4a468632-ae10-48f7-ab12-06aafca0130d_calibrate_994765_57.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/862176bb-9c7a-4f60-8377-54be2678340e_characterizeImage_994765_57.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/862176bb-9c7a-4f60-8377-54be2678340e_characterizeImage_994765_57.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e25a7336f1c8494a8914024c2739e4e4 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 1f4ce6dc4caf34ed3afcc74b372ae173 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f453bd8f-58e9-471a-8e6e-90ab8f74554d_calibrate_994766_7.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f453bd8f-58e9-471a-8e6e-90ab8f74554d_calibrate_994766_7.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/df761ce4-748c-484f-8194-987e91cb2a2d_characterizeImage_994766_7.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/df761ce4-748c-484f-8194-987e91cb2a2d_characterizeImage_994766_7.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 595 try 1 failed parsl.dataflow.dflow ERROR: Task 595 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 17db99cb6ac6666c2eb555e4f539866b with result from task 595 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 596 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 596 try 0 failed parsl.dataflow.dflow INFO: Task 596 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/13e75f7c-c0ed-46bc-a007-72e965a906c4_calibrate_994639_2.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/13e75f7c-c0ed-46bc-a007-72e965a906c4_calibrate_994639_2.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b997cb63-1bca-4323-9076-00289b72232c_characterizeImage_994639_2.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b997cb63-1bca-4323-9076-00289b72232c_characterizeImage_994639_2.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fa2809f1491090837fe7cc4cd7abd65 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 18708103614cdef0c160b46dbd3fa3b8 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c1be9e28-f229-4575-af6a-0ae01206ff97_calibrate_994767_24.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c1be9e28-f229-4575-af6a-0ae01206ff97_calibrate_994767_24.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fa9d003e-e592-4347-9100-7dc62ed466a5_characterizeImage_994767_24.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fa9d003e-e592-4347-9100-7dc62ed466a5_characterizeImage_994767_24.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98233d33a3df4e18a704917294f84304 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash c7c302d0dcfc8b3dff5e00c9228d182a parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/281b0927-cce3-4063-8c65-31170fd7ded5_calibrate_994767_57.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/281b0927-cce3-4063-8c65-31170fd7ded5_calibrate_994767_57.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8824782c-fce9-4e1f-b880-4b4091a1d20f_characterizeImage_994767_57.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8824782c-fce9-4e1f-b880-4b4091a1d20f_characterizeImage_994767_57.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd835953ab4e1f5e25d8a445c1d54de7 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash f00d469ccf33caa0aebc950a2d008de8 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ed29c0ea-ba96-4fb1-a767-016a3df73cc6_calibrate_994766_19.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ed29c0ea-ba96-4fb1-a767-016a3df73cc6_calibrate_994766_19.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a50a4ee7-2011-49ea-addc-7ad38ecc08ef_characterizeImage_994766_19.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a50a4ee7-2011-49ea-addc-7ad38ecc08ef_characterizeImage_994766_19.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7c51c0c68fac9647e260f9b964fb2e9 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 6e8f5b2dae81312b7b108254ee8b0f63 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6600400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d7460846-a5b7-442a-9abf-caf9321e5f27_calibrate_994767_55.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d7460846-a5b7-442a-9abf-caf9321e5f27_calibrate_994767_55.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fb45ea79-9132-4051-ba2c-4d049b154b69_characterizeImage_994767_55.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fb45ea79-9132-4051-ba2c-4d049b154b69_characterizeImage_994767_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:36 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5274e808b91ea2a360534eb5829ddc6 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash ffb766eff9e36c11c08825ef057ed4a6 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0a8a227a-8d92-45fd-b1d9-323b778e1ad5_calibrate_994766_52.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0a8a227a-8d92-45fd-b1d9-323b778e1ad5_calibrate_994766_52.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e91733ba-c9df-4351-9a1d-0e4fe56ad658_characterizeImage_994766_52.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e91733ba-c9df-4351-9a1d-0e4fe56ad658_characterizeImage_994766_52.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a51dea19e206eea545bbce58278c48b with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 3caa0602b1a2e5f21dc9c288faf5527f parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/313b140c-3bf9-4fc9-962a-e967ab4de04d_calibrate_994765_6.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/313b140c-3bf9-4fc9-962a-e967ab4de04d_calibrate_994765_6.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/fa93fab6-e60a-4b8f-b68b-e0aea60292ad_characterizeImage_994765_6.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/fa93fab6-e60a-4b8f-b68b-e0aea60292ad_characterizeImage_994765_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:37 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94fa62c313de2afebd1ae0fcd59fad5a with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash ffb40751c3b6179f9744812ce7cf10ec parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/788e83af-d439-401d-8898-8ef14a9eaac0_calibrate_994767_14.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/788e83af-d439-401d-8898-8ef14a9eaac0_calibrate_994767_14.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/30bc65c9-5e7e-4763-9f96-4ff7facbc830_characterizeImage_994767_14.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/30bc65c9-5e7e-4763-9f96-4ff7facbc830_characterizeImage_994767_14.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ad7a711accb650d793cf7212e52a21a with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 77c22736dd57c8071b2a8c5f8462669b parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8f974d6f-da2e-4d52-85bf-6e64430d7d02_calibrate_994767_30.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8f974d6f-da2e-4d52-85bf-6e64430d7d02_calibrate_994767_30.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/be01c5a7-f01d-43fb-96ef-207f2bbfbacc_characterizeImage_994767_30.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/be01c5a7-f01d-43fb-96ef-207f2bbfbacc_characterizeImage_994767_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:37 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aba9dff45f86fb802ab164cf739c91e9 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 28defee3178d6a1c2c3dca548fe67ca8 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0bcc6c94-9777-47ac-8b92-51227ba50ad3_calibrate_994766_59.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0bcc6c94-9777-47ac-8b92-51227ba50ad3_calibrate_994766_59.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/be80aebe-6111-4735-81a6-999bb8e775b0_characterizeImage_994766_59.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/be80aebe-6111-4735-81a6-999bb8e775b0_characterizeImage_994766_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:38 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 583 try 1 failed parsl.dataflow.dflow ERROR: Task 583 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry ea48ae85d1342e4694a5932b5aa01265 with result from task 583 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 584 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 584 try 0 failed parsl.dataflow.dflow INFO: Task 584 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/13fbdbf4-a568-4b82-b195-b5f7a454892f_calibrate_994640_2.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/13fbdbf4-a568-4b82-b195-b5f7a454892f_calibrate_994640_2.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b7340884-dcdf-41e2-bb15-aa1a08d29bcf_characterizeImage_994640_2.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b7340884-dcdf-41e2-bb15-aa1a08d29bcf_characterizeImage_994640_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:38 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 269b19cfddb2621e32d439501d283e6b with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash a26daee3b3309ab6c0b02d215ae69155 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b87060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/7ea8637e-a7a3-43a5-9300-4dd0ebd3c1ee_calibrate_994765_25.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/7ea8637e-a7a3-43a5-9300-4dd0ebd3c1ee_calibrate_994765_25.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/17526b9f-7075-490f-a988-09f0c5943b35_characterizeImage_994765_25.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/17526b9f-7075-490f-a988-09f0c5943b35_characterizeImage_994765_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:38 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4855b68326bb96a88e3597f3783d672a with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash c32c2b5c5480f06c3977b691b44620b4 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66013a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1267df00-9eac-45c3-a88d-bc3ff890ea43_calibrate_994767_31.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1267df00-9eac-45c3-a88d-bc3ff890ea43_calibrate_994767_31.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/45f6c1a2-4c41-401a-bcfe-df7211ec983d_characterizeImage_994767_31.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/45f6c1a2-4c41-401a-bcfe-df7211ec983d_characterizeImage_994767_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:39 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 402a137e3ca1151e17c2af66ce85fe74 with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash a2cb84a1fa1880f7c23fa37a75a7e173 parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65047c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8b5ad807-af06-4631-9519-3c920d523d7c_calibrate_994766_39.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8b5ad807-af06-4631-9519-3c920d523d7c_calibrate_994766_39.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/6c707e79-3404-48fd-a09f-f9effae741bc_characterizeImage_994766_39.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/6c707e79-3404-48fd-a09f-f9effae741bc_characterizeImage_994766_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:39 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b114e0483e75a6bb5abefb6df81243f with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash d74df2def58f5313fb8fe25db9dab55b parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c4923ac2-5777-49c6-bf5b-b1667d6060e6_calibrate_994765_34.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c4923ac2-5777-49c6-bf5b-b1667d6060e6_calibrate_994765_34.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/278fa85a-6a39-46c7-8d0d-ab81047db64a_characterizeImage_994765_34.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/278fa85a-6a39-46c7-8d0d-ab81047db64a_characterizeImage_994765_34.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23d4a9df9e6a5a10eceeeac86166d02c with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 632e3157ed459c5fd694b7cc141c3877 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/78dfa728-5c5d-40dd-81e1-6d2d1697aff7_calibrate_994765_41.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/78dfa728-5c5d-40dd-81e1-6d2d1697aff7_calibrate_994765_41.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/9fb224e5-2b04-459f-b2d1-5128ca4d35ed_characterizeImage_994765_41.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/9fb224e5-2b04-459f-b2d1-5128ca4d35ed_characterizeImage_994765_41.stderr parsl.dataflow.dflow INFO: Task 964 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a2bca73719045d51e1f4c8a49d0cb59 with result from task 964 parsl.dataflow.dflow DEBUG: Task 964 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 965 has memoization hash 8ccae2cfa65db9c291c1fb43230bc19b parsl.dataflow.memoization INFO: Task 965 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 965 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/153e3e89-9c9b-4af5-bd69-9fc862b791e6_calibrate_994766_45.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/153e3e89-9c9b-4af5-bd69-9fc862b791e6_calibrate_994766_45.stderr parsl.dataflow.dflow INFO: Standard output for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/97d1b19d-dbf8-4030-9ffb-3b8514f21233_characterizeImage_994766_45.stdout parsl.dataflow.dflow INFO: Standard error for task 964 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/97d1b19d-dbf8-4030-9ffb-3b8514f21233_characterizeImage_994766_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:40 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d97c602b35d5ba954322576f31c08c62 with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 7f8f608a20ba4933ed4ff0efd679847c parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65060c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/8afac479-8583-4bbc-98f0-8a4ebd7a8f36_calibrate_994765_18.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/8afac479-8583-4bbc-98f0-8a4ebd7a8f36_calibrate_994765_18.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b7b9dbb5-ffe2-4bb5-b208-63ffae20188e_characterizeImage_994765_18.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b7b9dbb5-ffe2-4bb5-b208-63ffae20188e_characterizeImage_994765_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:40 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13c64a1b3de4d1319ff0f48a3aceef2f with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash 1f702432f988dabe042effb75d130484 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66502c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/6fd16fb6-9d59-4993-94cb-65bcee574ab0_calibrate_994765_31.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/6fd16fb6-9d59-4993-94cb-65bcee574ab0_calibrate_994765_31.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/76b6a8b7-49c6-4424-8e48-5e6e3454ab06_characterizeImage_994765_31.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/76b6a8b7-49c6-4424-8e48-5e6e3454ab06_characterizeImage_994765_31.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca92604bed00eb643369490c6ecb372f with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 22f482dd6683ad5e356b94ad43befdc6 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/58c69f05-94a6-4ac0-a4fa-159d5016141f_calibrate_994767_11.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/58c69f05-94a6-4ac0-a4fa-159d5016141f_calibrate_994767_11.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/27c87c3e-cf08-43c5-92a9-d52a19d5f28c_characterizeImage_994767_11.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/27c87c3e-cf08-43c5-92a9-d52a19d5f28c_characterizeImage_994767_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:41 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b942cd203baeb883cd4d960dffa7354 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash dc4c6eee58a80ee9f079cab125e41061 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b85760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f58a330c-e3b7-46bf-9026-774d718b0c6a_calibrate_994766_23.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f58a330c-e3b7-46bf-9026-774d718b0c6a_calibrate_994766_23.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/be8fc17c-d578-4566-b1c2-eaa94c2351f8_characterizeImage_994766_23.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/be8fc17c-d578-4566-b1c2-eaa94c2351f8_characterizeImage_994766_23.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45b44074824b7adab8429ecb9b182877 with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 1ad2b642445f24e852e11d5215739272 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/53f6f463-1b36-4bc7-a167-172b6f6e852a_calibrate_994765_7.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/53f6f463-1b36-4bc7-a167-172b6f6e852a_calibrate_994765_7.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/996e02ab-de4d-4835-945b-226f07041bb3_characterizeImage_994765_7.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/996e02ab-de4d-4835-945b-226f07041bb3_characterizeImage_994765_7.stderr parsl.dataflow.dflow INFO: Task 1096 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6719d0e190ff857546d5d72ad0c30b5f with result from task 1096 parsl.dataflow.dflow DEBUG: Task 1096 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1097 has memoization hash 6f92955dd37af412259a6c9b853e5e9a parsl.dataflow.memoization INFO: Task 1097 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1097 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/60c98210-d613-42c9-826e-f7eb90fd0d10_calibrate_994765_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/60c98210-d613-42c9-826e-f7eb90fd0d10_calibrate_994765_23.stderr parsl.dataflow.dflow INFO: Standard output for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d58975b9-e486-4999-971d-37ed98983e80_characterizeImage_994765_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1096 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d58975b9-e486-4999-971d-37ed98983e80_characterizeImage_994765_23.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac38784094ca88ed93a64304704c35f9 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 5ec843f393c9df1b0b9fa5c984fb1b11 parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6600d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/e710dd07-232d-45cd-a7e7-288d7e419fc3_calibrate_994765_19.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/e710dd07-232d-45cd-a7e7-288d7e419fc3_calibrate_994765_19.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/4beb5e42-ae90-446a-a7f5-4bc1549682d3_characterizeImage_994765_19.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/4beb5e42-ae90-446a-a7f5-4bc1549682d3_characterizeImage_994765_19.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0825a24751e227d2ea4106e7a4b176d5 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 59bcf7d41ae8386c69a8620e3e7f20cd parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0aa80de6-af7d-4bc6-9ec2-85da45f5ff03_calibrate_994766_18.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0aa80de6-af7d-4bc6-9ec2-85da45f5ff03_calibrate_994766_18.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/b6f85c45-7d01-47ec-96b2-a7bb1af2a1fa_characterizeImage_994766_18.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/b6f85c45-7d01-47ec-96b2-a7bb1af2a1fa_characterizeImage_994766_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:41 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd5103fd4d51bcffdab14c7d331e2f42 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 088597bbdc3e6a4520f72fdc2d8e4f4b parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/7bf3ac5f-2e3a-4f7e-85a8-c7743b313bb4_calibrate_994766_36.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/7bf3ac5f-2e3a-4f7e-85a8-c7743b313bb4_calibrate_994766_36.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c1b80f70-805e-423a-a86e-eeb9f146c1e2_characterizeImage_994766_36.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c1b80f70-805e-423a-a86e-eeb9f146c1e2_characterizeImage_994766_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:41 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0456cce8c4d63c12c39f2511599037ea with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash 0ea8bd27321f39efd05d6d8b0952e641 parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63925c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f6df9893-416b-41dc-8477-614b41793f1e_calibrate_994765_48.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f6df9893-416b-41dc-8477-614b41793f1e_calibrate_994765_48.stderr parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d6eefd11-0721-4ba3-90ba-80f377a0ddc4_characterizeImage_994765_48.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d6eefd11-0721-4ba3-90ba-80f377a0ddc4_characterizeImage_994765_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:42 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 955 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83a009936e534668cb7564bbc075ff18 with result from task 955 parsl.dataflow.dflow DEBUG: Task 955 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 956 has memoization hash 260d7494fa2b0af5a9629f1bc03bf2a6 parsl.dataflow.memoization INFO: Task 956 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 956 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/054d1ec5-f725-4401-a354-1a34e12c26c4_calibrate_994765_21.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/054d1ec5-f725-4401-a354-1a34e12c26c4_calibrate_994765_21.stderr parsl.dataflow.dflow INFO: Standard output for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8b4cbb2c-78ea-4ec9-b8e8-9f60d88ec480_characterizeImage_994765_21.stdout parsl.dataflow.dflow INFO: Standard error for task 955 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8b4cbb2c-78ea-4ec9-b8e8-9f60d88ec480_characterizeImage_994765_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:42 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d1d5f1bd415964a744db117f8a0b35c with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 44be1a929d38b9fd62ff433c0f7b4e17 parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2656270a-e7a7-4e40-bd83-2bd0b3676570_calibrate_994767_9.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2656270a-e7a7-4e40-bd83-2bd0b3676570_calibrate_994767_9.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/a2b89b58-c51e-4250-86a6-009a90d37b56_characterizeImage_994767_9.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/a2b89b58-c51e-4250-86a6-009a90d37b56_characterizeImage_994767_9.stderr parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2722ed0b3a55f2536afc0b1cdacfd62d with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 1ebe7fae65c84f5876d17ddea4436064 parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6506b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/76b134bb-25d8-40c5-aa3b-7f07125d2af3_calibrate_994765_22.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/76b134bb-25d8-40c5-aa3b-7f07125d2af3_calibrate_994765_22.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8e4ee58f-eca9-4187-8bd4-b273d5bfbf20_characterizeImage_994765_22.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8e4ee58f-eca9-4187-8bd4-b273d5bfbf20_characterizeImage_994765_22.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c106e286ad3089d7cc3856e0f8157d33 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 1374540f4975b7f6db52205d93f5b786 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/e0d69843-dc27-4d33-9eee-569fee7625d8_calibrate_994767_21.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/e0d69843-dc27-4d33-9eee-569fee7625d8_calibrate_994767_21.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/de1b72a2-fa6c-463a-8613-b50d50d8041f_characterizeImage_994767_21.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/de1b72a2-fa6c-463a-8613-b50d50d8041f_characterizeImage_994767_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:43 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac5da1fe87b7f6e0fb0dca8c43248205 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash dc99bc368f9b92ca6c9b7c1673bc80a7 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b87ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ee3215b7-5903-4f1b-a03f-eaa6bdba0bb9_calibrate_994767_27.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ee3215b7-5903-4f1b-a03f-eaa6bdba0bb9_calibrate_994767_27.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/71e6da64-dce3-40f2-81fe-cc41bab94d1e_characterizeImage_994767_27.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/71e6da64-dce3-40f2-81fe-cc41bab94d1e_characterizeImage_994767_27.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0af3730abc996e453b81c2c8487dcca1 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 64f06a13f44d27ad9e69ac15979808e3 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6f13856b-adc8-4793-b0a0-70dcf742e270_calibrate_994766_16.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6f13856b-adc8-4793-b0a0-70dcf742e270_calibrate_994766_16.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c7975d08-5394-491d-99aa-da315035f3da_characterizeImage_994766_16.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c7975d08-5394-491d-99aa-da315035f3da_characterizeImage_994766_16.stderr parsl.dataflow.dflow INFO: Task 973 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08bc30a0d27fb76a0544a7223a8bac3c with result from task 973 parsl.dataflow.dflow DEBUG: Task 973 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 974 has memoization hash 1e3b877e97e657b9618045d8d2b1c010 parsl.dataflow.memoization INFO: Task 974 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 974 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b1744763-105b-4162-9fe3-4f2777beb982_calibrate_994767_6.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b1744763-105b-4162-9fe3-4f2777beb982_calibrate_994767_6.stderr parsl.dataflow.dflow INFO: Standard output for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0e8b0a1f-3fe3-4ede-9866-b2faa225d5c1_characterizeImage_994767_6.stdout parsl.dataflow.dflow INFO: Standard error for task 973 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0e8b0a1f-3fe3-4ede-9866-b2faa225d5c1_characterizeImage_994767_6.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51af2f7821ec11877b3e8aa3696118cd with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 9c6ade5fefefe2d2164fbfb18620ae16 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/5aea1c6e-53ff-4b41-b051-34a7a85e90e1_calibrate_994765_27.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/5aea1c6e-53ff-4b41-b051-34a7a85e90e1_calibrate_994765_27.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a16849d7-356f-4dfa-87f9-3a31f184a365_characterizeImage_994765_27.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a16849d7-356f-4dfa-87f9-3a31f184a365_characterizeImage_994765_27.stderr parsl.dataflow.dflow INFO: Task 1015 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 790781215f24cc6fd2a80c64db94d6ef with result from task 1015 parsl.dataflow.dflow DEBUG: Task 1015 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1016 has memoization hash 2feacbcb09bada67a8f2637f0a3df198 parsl.dataflow.memoization INFO: Task 1016 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1016 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/e19bbdc3-99b5-460c-8e2d-878504a44744_calibrate_994767_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/e19bbdc3-99b5-460c-8e2d-878504a44744_calibrate_994767_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/856ca72b-949e-4c20-b533-78fb7f41c967_characterizeImage_994767_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1015 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/856ca72b-949e-4c20-b533-78fb7f41c967_characterizeImage_994767_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:43 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cf324825a590b7b543c84bfa5f78a6f with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash 5009df8164c5fca88cb294cea66466aa parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6507380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/15856891-ae06-445f-99a6-a502e1fe7bf5_calibrate_994766_25.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/15856891-ae06-445f-99a6-a502e1fe7bf5_calibrate_994766_25.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/103a32e2-ebdb-47ed-97d4-f0644d2c6dff_characterizeImage_994766_25.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/103a32e2-ebdb-47ed-97d4-f0644d2c6dff_characterizeImage_994766_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:44 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fead8dd5762df4effdb885765115bc5 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash b3b4c6a2dc09a8a7cc35c9f6b1388ba9 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/be68004b-4cf7-43b2-a4f7-e6831a05d729_calibrate_994767_4.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/be68004b-4cf7-43b2-a4f7-e6831a05d729_calibrate_994767_4.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/53947b9a-cfff-438b-aff7-a2e7d53bd0c6_characterizeImage_994767_4.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/53947b9a-cfff-438b-aff7-a2e7d53bd0c6_characterizeImage_994767_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:45 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fd28d6e1b5adf78c9cfc702f4d1f7d0 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 7611d61860aab6406519eca932ae96c5 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/447f2fee-af9a-49a0-9663-63d9e9235dfc_calibrate_994766_60.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/447f2fee-af9a-49a0-9663-63d9e9235dfc_calibrate_994766_60.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2421ba9d-8a04-4d6d-ba4a-b249fd90fd81_characterizeImage_994766_60.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2421ba9d-8a04-4d6d-ba4a-b249fd90fd81_characterizeImage_994766_60.stderr parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2cbaa5f0a353ac358624401a1bd3d29 with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash e63ca1ddd6ded40367d8b7d5c3fcac7f parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/d8ebb432-b730-4eb3-93ae-97e85eca07ac_calibrate_994766_22.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/d8ebb432-b730-4eb3-93ae-97e85eca07ac_calibrate_994766_22.stderr parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/00587d06-a18c-482a-a772-ad5ad4c442f9_characterizeImage_994766_22.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/00587d06-a18c-482a-a772-ad5ad4c442f9_characterizeImage_994766_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:45 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4864e005fe8f1007b965c544a4a0c08 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash ff68618ed33b0d817ded9fdb1d115dd7 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/fdccfb17-6578-4621-9e65-23f4cf8aa8a6_calibrate_994767_53.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/fdccfb17-6578-4621-9e65-23f4cf8aa8a6_calibrate_994767_53.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/9c545647-93df-4a6d-949d-6755eefee481_characterizeImage_994767_53.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/9c545647-93df-4a6d-949d-6755eefee481_characterizeImage_994767_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:45 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 232dd4caca653cce34611363ea030ee8 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 57b327d34d0383b5ffd5ce43f8838b9f parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f2b7962a-0ef6-4709-a013-be0c847912ee_calibrate_994765_39.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f2b7962a-0ef6-4709-a013-be0c847912ee_calibrate_994765_39.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b3fec195-97e4-454f-bd4f-b832115922b5_characterizeImage_994765_39.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b3fec195-97e4-454f-bd4f-b832115922b5_characterizeImage_994765_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:46 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5539341f89a323ba9c225fa051caa6c with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 85ff2836e7c24b59231dbebe950dc350 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a8a7fdd1-f8e1-43d7-a4ce-e1cae5825b7b_calibrate_994765_38.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a8a7fdd1-f8e1-43d7-a4ce-e1cae5825b7b_calibrate_994765_38.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/752867fe-b703-4718-9716-acd73ea20183_characterizeImage_994765_38.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/752867fe-b703-4718-9716-acd73ea20183_characterizeImage_994765_38.stderr parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36c10c0ec012e42ca978f0c0d3461752 with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash de308c0b33535c4a92b28112384ace90 parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/9e5c93e2-0731-4793-a72d-d01786d43999_calibrate_994766_40.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/9e5c93e2-0731-4793-a72d-d01786d43999_calibrate_994766_40.stderr parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5cad8192-20c4-4049-aebd-8512698ddc0a_characterizeImage_994766_40.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5cad8192-20c4-4049-aebd-8512698ddc0a_characterizeImage_994766_40.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d557eadb3d885f54b5af0de2e14776ce with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash d9c02ea094f5f5a08c2881a672c9b74f parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64877e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/bcfc2265-a6e8-4925-ba3b-e1a6382b5feb_calibrate_994767_46.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/bcfc2265-a6e8-4925-ba3b-e1a6382b5feb_calibrate_994767_46.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/69d498d3-aabe-4186-935d-1197a9345c9d_characterizeImage_994767_46.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/69d498d3-aabe-4186-935d-1197a9345c9d_characterizeImage_994767_46.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa0bc051e1e28e8cd94ce547aa9b23b5 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 0303759a025398b144983105ffe3dd6c parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2a62b54a-b4ea-4240-8a7a-982660cf3dbb_calibrate_994765_52.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2a62b54a-b4ea-4240-8a7a-982660cf3dbb_calibrate_994765_52.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f7258449-3982-4313-96d2-3a60b3502a4c_characterizeImage_994765_52.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f7258449-3982-4313-96d2-3a60b3502a4c_characterizeImage_994765_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:46 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f4550b36df7e5491cf9d6286cfa53f2 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 2450fb3dcf86de8382d398c5a6e2552c parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8d99af81-c39e-4683-8c95-91fdef59c5fb_calibrate_994767_28.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8d99af81-c39e-4683-8c95-91fdef59c5fb_calibrate_994767_28.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0fd6d13c-0b81-4963-874a-3a3aac492273_characterizeImage_994767_28.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0fd6d13c-0b81-4963-874a-3a3aac492273_characterizeImage_994767_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:47 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2480c60ef97d9b929228d1936886291b with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 126dd9186653dfc162c1f46a16167d67 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e54e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/112b98c6-ac16-4a59-ad3e-9fa08038c4cc_calibrate_994767_34.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/112b98c6-ac16-4a59-ad3e-9fa08038c4cc_calibrate_994767_34.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/88039974-223a-4da6-8ede-b0388fea1d0b_characterizeImage_994767_34.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/88039974-223a-4da6-8ede-b0388fea1d0b_characterizeImage_994767_34.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8231195fca1bbee2730752bf628bcce5 with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash 3caaa08dca41c5c950caad5a13e466b3 parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63916c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/206c150f-99ae-43f6-acb0-ecd11f8fd61c_calibrate_994765_44.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/206c150f-99ae-43f6-acb0-ecd11f8fd61c_calibrate_994765_44.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ca86d385-d072-430a-bcb6-db791875473f_characterizeImage_994765_44.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ca86d385-d072-430a-bcb6-db791875473f_characterizeImage_994765_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:47 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5643bc25f63cd0c72aafc4071d600d32 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash 8b8303bf7226be7b3db132046eefe7b0 parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6487b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/87d55d4b-d458-4a02-a71c-4a52e94c4247_calibrate_994765_37.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/87d55d4b-d458-4a02-a71c-4a52e94c4247_calibrate_994765_37.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/85cc91ff-4723-4451-aa26-b0e1c5557cfd_characterizeImage_994765_37.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/85cc91ff-4723-4451-aa26-b0e1c5557cfd_characterizeImage_994765_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:47 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7d9b3df96f96b08ff99e0298925e824 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash b5b993016436fb839e34d11ff800ab3a parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/e749c11d-c4ad-45be-8172-e694922e4648_calibrate_994766_53.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/e749c11d-c4ad-45be-8172-e694922e4648_calibrate_994766_53.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2de9f952-62ed-48c2-a3f4-555ff942ea9f_characterizeImage_994766_53.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2de9f952-62ed-48c2-a3f4-555ff942ea9f_characterizeImage_994766_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:48 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 1105 try 1 failed parsl.dataflow.dflow ERROR: Task 1105 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 670bdac47e3c118575dcb15a0abb4599 with result from task 1105 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 1106 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 1106 try 0 failed parsl.dataflow.dflow INFO: Task 1106 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/6528ee46-c809-4ade-b9be-269b2beb9f91_calibrate_994638_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/6528ee46-c809-4ade-b9be-269b2beb9f91_calibrate_994638_2.stderr parsl.dataflow.dflow INFO: Standard output for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9d3d106b-0aca-4ac0-ad90-088f5cd478d7_characterizeImage_994638_2.stdout parsl.dataflow.dflow INFO: Standard error for task 1105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9d3d106b-0aca-4ac0-ad90-088f5cd478d7_characterizeImage_994638_2.stderr parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 131c4639e08a796bc4804806a91da93b with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash c3b2772fef37ae38cccfe99ef9ee2b4a parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6392f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d24220af-6d9f-4c72-b216-d1f73d4017d3_calibrate_994767_42.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d24220af-6d9f-4c72-b216-d1f73d4017d3_calibrate_994767_42.stderr parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/e5061510-ea4d-463b-a166-d283b5bf511c_characterizeImage_994767_42.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/e5061510-ea4d-463b-a166-d283b5bf511c_characterizeImage_994767_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:48 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9381ea68baf78a0a02a038b1daf10432 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash ada352ecc7a42daedd10a5bde4877624 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a7380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/297c5151-dbc0-4d53-aace-e061bc6bc8b6_calibrate_994767_18.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/297c5151-dbc0-4d53-aace-e061bc6bc8b6_calibrate_994767_18.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/52145c6d-1bea-40fd-8fcd-cbe82c3591d6_characterizeImage_994767_18.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/52145c6d-1bea-40fd-8fcd-cbe82c3591d6_characterizeImage_994767_18.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8307334fabc7afc493d9fe24b15b121e with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 42406d75e32774eb7122fdc89992c66a parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ae5e941d-6a2f-4813-953d-db84752bc9c4_calibrate_994767_29.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ae5e941d-6a2f-4813-953d-db84752bc9c4_calibrate_994767_29.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fddfe5b1-103c-4ad1-971f-6f795781e444_characterizeImage_994767_29.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fddfe5b1-103c-4ad1-971f-6f795781e444_characterizeImage_994767_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:48 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c35db0320d6b5142b6ad4c0318e8b2c6 with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 2a64085943f68ce10d74da0ce3035640 parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/bbff916a-2c69-44dd-8320-5ea54974bb8f_calibrate_994767_5.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/bbff916a-2c69-44dd-8320-5ea54974bb8f_calibrate_994767_5.stderr parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/dca4f1ba-4451-4053-bf79-1635a390d6f8_characterizeImage_994767_5.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/dca4f1ba-4451-4053-bf79-1635a390d6f8_characterizeImage_994767_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:48 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09d8f845c136bdb1efa1c4338d94dec0 with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash e65e63a90c24b26b1456953ebd417059 parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65058a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/d4279bbc-b62a-4d9d-85ec-f22e3d48f69c_calibrate_994766_32.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/d4279bbc-b62a-4d9d-85ec-f22e3d48f69c_calibrate_994766_32.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/147d1f36-cdeb-4f1b-abdb-8730758decdf_characterizeImage_994766_32.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/147d1f36-cdeb-4f1b-abdb-8730758decdf_characterizeImage_994766_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:49 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7729559f48603683b4e731248a7ffa62 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 088afe3ff33d316003d99231ef3ccaf7 parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66509a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/33c7a2ec-1ec6-478f-9d6e-81efa6be96e0_calibrate_994767_54.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/33c7a2ec-1ec6-478f-9d6e-81efa6be96e0_calibrate_994767_54.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2ff4ad02-3989-4423-ad59-71b3cfe7a256_characterizeImage_994767_54.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2ff4ad02-3989-4423-ad59-71b3cfe7a256_characterizeImage_994767_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:49 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eba8bffb09159de60fda4d7d7029c265 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 59c1aa7f7cd53b4b18f64e2c5f012d5d parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ddb29960-86e9-44ac-bf5c-178331d06d7c_calibrate_994767_32.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ddb29960-86e9-44ac-bf5c-178331d06d7c_calibrate_994767_32.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/98637530-2b75-4eb1-ae70-f92fc430b7df_characterizeImage_994767_32.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/98637530-2b75-4eb1-ae70-f92fc430b7df_characterizeImage_994767_32.stderr parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97483f7b040ae9d90ab5013f1a62fc00 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash 34fb21cf3952f0b629b577bc85735679 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/40ca6169-d806-4f0a-8113-5f98f183f8e6_calibrate_994767_36.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/40ca6169-d806-4f0a-8113-5f98f183f8e6_calibrate_994767_36.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/919cf315-3584-4e95-8b2e-7fd38e8a2282_characterizeImage_994767_36.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/919cf315-3584-4e95-8b2e-7fd38e8a2282_characterizeImage_994767_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:50 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4529521fd7cdb23bf1e475de3e5ff91 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 9ebfe3135fb76bcd23f56da8a4628d29 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ff536819-e375-4b4a-ba95-30b14eb371a0_calibrate_994766_56.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ff536819-e375-4b4a-ba95-30b14eb371a0_calibrate_994766_56.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2d730346-779d-4d10-967e-627f4a1889d2_characterizeImage_994766_56.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2d730346-779d-4d10-967e-627f4a1889d2_characterizeImage_994766_56.stderr parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9c5dc142898e5671265004975b4629c with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash 7fed6937667d93e73e9327d614822188 parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/02674561-56f1-40c9-b524-f173b1a28384_calibrate_994765_28.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/02674561-56f1-40c9-b524-f173b1a28384_calibrate_994765_28.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b8661903-be6e-44eb-a3fa-a51429affe2c_characterizeImage_994765_28.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b8661903-be6e-44eb-a3fa-a51429affe2c_characterizeImage_994765_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:50 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4648eda29cbf5fe25c1e1fa889691496 with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash dcec12c61e2067cd70170e9dcfd14052 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/073d30e0-4ad6-4a1d-b121-0e313156018f_calibrate_994765_20.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/073d30e0-4ad6-4a1d-b121-0e313156018f_calibrate_994765_20.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8a192b74-8f48-4dd3-8d95-aafd0ee49930_characterizeImage_994765_20.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8a192b74-8f48-4dd3-8d95-aafd0ee49930_characterizeImage_994765_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:50 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 997 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5d39520420570591316d74fcb4b0831 with result from task 997 parsl.dataflow.dflow DEBUG: Task 997 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 998 has memoization hash a2f2a17f8a1d76f010cf9cd0d5b6b49e parsl.dataflow.memoization INFO: Task 998 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 998 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/369cc22c-0514-45a4-8c70-3060e705a17b_calibrate_994765_43.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/369cc22c-0514-45a4-8c70-3060e705a17b_calibrate_994765_43.stderr parsl.dataflow.dflow INFO: Standard output for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/48647308-0fc5-4d8e-8154-3711de360b88_characterizeImage_994765_43.stdout parsl.dataflow.dflow INFO: Standard error for task 997 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/48647308-0fc5-4d8e-8154-3711de360b88_characterizeImage_994765_43.stderr parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a76206fe0e26b528af82a188b3a4aae with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 92c5a63f21e57ecb864f6a216afdc259 parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66527a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/1e4f7f36-047d-491c-95b2-feed56294ca7_calibrate_994766_14.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/1e4f7f36-047d-491c-95b2-feed56294ca7_calibrate_994766_14.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/74ad22a1-4185-4bf7-8e5a-85972e528359_characterizeImage_994766_14.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/74ad22a1-4185-4bf7-8e5a-85972e528359_characterizeImage_994766_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:51 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba4350bc5c89876c7b157f2089ad5227 with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 6163bdd5dfa18fac26501b1fbf896fb4 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3824aaf5-d001-45c7-a267-7240d2dd61d9_calibrate_994766_20.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3824aaf5-d001-45c7-a267-7240d2dd61d9_calibrate_994766_20.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/04fe4f2b-a462-4d9a-a353-0d3e031dc134_characterizeImage_994766_20.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/04fe4f2b-a462-4d9a-a353-0d3e031dc134_characterizeImage_994766_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:51 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:52 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a3df981f82fcf0b575cee180c9dd409 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 8b46a067b509f8834db2c3ed0bcd2449 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e67a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b76bba7d-4ba6-45d5-bc89-996f6b7272b0_calibrate_994767_37.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b76bba7d-4ba6-45d5-bc89-996f6b7272b0_calibrate_994767_37.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/304d2c1e-124c-4f83-8008-4f10093e9a3f_characterizeImage_994767_37.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/304d2c1e-124c-4f83-8008-4f10093e9a3f_characterizeImage_994767_37.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b981a82266165e9aab9d7416d8a3df8 with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 94354bc7d8a8ff82c6bf4acf813c2d9f parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c7192578-4fcd-45b2-95e9-9aaa8be648d6_calibrate_994767_20.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c7192578-4fcd-45b2-95e9-9aaa8be648d6_calibrate_994767_20.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/f8d2b92d-abf3-452e-8813-95a698a3a2a7_characterizeImage_994767_20.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/f8d2b92d-abf3-452e-8813-95a698a3a2a7_characterizeImage_994767_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:52 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dab2aa1155fef4d34a98413808e9dec with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 42555677bb39d8b7469db8d9d89f9664 parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64beb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1b2a9be1-94a3-418b-ae63-1b24394c026b_calibrate_994767_23.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1b2a9be1-94a3-418b-ae63-1b24394c026b_calibrate_994767_23.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2bd03605-3947-4e60-a262-eee9e5c81f88_characterizeImage_994767_23.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2bd03605-3947-4e60-a262-eee9e5c81f88_characterizeImage_994767_23.stderr parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28af9932da73fcf00298d3f022bf22ab with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash 5e29febd9bdc30adee9c0374f12610b6 parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ea9685bd-a71b-4114-a686-a24e46643823_calibrate_994766_62.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ea9685bd-a71b-4114-a686-a24e46643823_calibrate_994766_62.stderr parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/d2970fc3-e43d-4e47-bbf7-22c5256eeca5_characterizeImage_994766_62.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/d2970fc3-e43d-4e47-bbf7-22c5256eeca5_characterizeImage_994766_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:53 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09e535754fbb0ea02f62f04a4a2d317f with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 06fe448163bbe009246ee40979568e8c parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/39830acb-b9c7-462b-bf09-1ed4765348c9_calibrate_994767_16.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/39830acb-b9c7-462b-bf09-1ed4765348c9_calibrate_994767_16.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4817a43b-a354-4e8e-a07b-3d7f9c413ab3_characterizeImage_994767_16.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4817a43b-a354-4e8e-a07b-3d7f9c413ab3_characterizeImage_994767_16.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b13b15d540dd7186d87893aa8a5784e1 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 8ffe9b1c09bc5594c85159706f32507e parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b57e2608-0e8d-4cec-bd01-d824bdf60be1_calibrate_994765_45.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b57e2608-0e8d-4cec-bd01-d824bdf60be1_calibrate_994765_45.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c948c518-16d8-4586-8389-0790dcd867aa_characterizeImage_994765_45.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c948c518-16d8-4586-8389-0790dcd867aa_characterizeImage_994765_45.stderr parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 506ad8943533ad2154b526b702acd362 with result from task 865 parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:53 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash fbe9b5e638a8cffbc9183df46046a224 parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8539292e-3ff7-4b01-a5d3-7bf98cd63d3e_calibrate_994766_29.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8539292e-3ff7-4b01-a5d3-7bf98cd63d3e_calibrate_994766_29.stderr parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/9ac183ae-ea5d-4800-88c6-937708a5521c_characterizeImage_994766_29.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/9ac183ae-ea5d-4800-88c6-937708a5521c_characterizeImage_994766_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:54 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1cea13fdc7c96e928d4dae834a97cca0 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 9d27e5cde400ba4ef75dd793b6dbaaf9 parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6651b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8c604ef-5e3f-4785-a7f9-d5b6732975d4_calibrate_994766_35.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8c604ef-5e3f-4785-a7f9-d5b6732975d4_calibrate_994766_35.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/092009d3-b0f8-4d69-98a1-512af2898b08_characterizeImage_994766_35.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/092009d3-b0f8-4d69-98a1-512af2898b08_characterizeImage_994766_35.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9795a8fe07cc6706d4e5bca519a72fee with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 76fa0c55a29c68854f34d046642d412e parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66019e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/74784a72-c643-442a-8969-e4f0d11604d2_calibrate_994767_8.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/74784a72-c643-442a-8969-e4f0d11604d2_calibrate_994767_8.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/43de96a2-e1be-44d4-b15b-75a79e13a43f_characterizeImage_994767_8.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/43de96a2-e1be-44d4-b15b-75a79e13a43f_characterizeImage_994767_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:54 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 979 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ada1340060250a49754aa9be57cbfb13 with result from task 979 parsl.dataflow.dflow DEBUG: Task 979 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 980 has memoization hash 1ce7dce47b723f56711bcd590fa92dcb parsl.dataflow.memoization INFO: Task 980 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 980 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f13b5ffb-b903-4386-9a66-c901e8246aa1_calibrate_994766_8.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f13b5ffb-b903-4386-9a66-c901e8246aa1_calibrate_994766_8.stderr parsl.dataflow.dflow INFO: Standard output for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/875f4239-c074-4b03-b815-599dd66f8292_characterizeImage_994766_8.stdout parsl.dataflow.dflow INFO: Standard error for task 979 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/875f4239-c074-4b03-b815-599dd66f8292_characterizeImage_994766_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:54 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ba585bc83ad94d73b0fea1b74a51f35 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash a704c8d07cdcb05a1131da1ff0fdb3af parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c4b34b89-1679-4e4c-9193-354f192d9624_calibrate_994767_17.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c4b34b89-1679-4e4c-9193-354f192d9624_calibrate_994767_17.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/129c75af-580a-4dfb-87c8-5ba3ad49522b_characterizeImage_994767_17.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/129c75af-580a-4dfb-87c8-5ba3ad49522b_characterizeImage_994767_17.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a80656f99130d4a496d0033330e61c38 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 4834ed578a62399ec4db3b8b2ac693ec parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6600720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/38210995-54ff-4d3b-bce2-425fa71f9c10_calibrate_994766_11.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/38210995-54ff-4d3b-bce2-425fa71f9c10_calibrate_994766_11.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a8359622-4990-4bae-bf87-5d9f1b973951_characterizeImage_994766_11.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a8359622-4990-4bae-bf87-5d9f1b973951_characterizeImage_994766_11.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df3a8d78ebe301183291835df7d390f7 with result from task 454 parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 9b9758c83313e357406af68be7fc1136 parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6652c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/8d2dcc2b-05a1-405b-961f-f6c7f52a1110_calibrate_994765_54.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/8d2dcc2b-05a1-405b-961f-f6c7f52a1110_calibrate_994765_54.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/15e52219-0dff-426e-91e0-090be7aacae0_characterizeImage_994765_54.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/15e52219-0dff-426e-91e0-090be7aacae0_characterizeImage_994765_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:55 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c78f372de130ce2941b216d1b69e331e with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash 5366b79f484ae8772b432390ce80dd57 parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63922a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8f221a0-209f-4d5c-91cd-3134e24c74a1_calibrate_994766_27.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8f221a0-209f-4d5c-91cd-3134e24c74a1_calibrate_994766_27.stderr parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/465073c6-35cb-4523-82f6-76f5ddfee818_characterizeImage_994766_27.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/465073c6-35cb-4523-82f6-76f5ddfee818_characterizeImage_994766_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 366 parsl.dataflow.strategy DEBUG: Executor multi has 366 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:14:55 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3412b380d31dd54b0bd62d09cfbe8577 with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4bca1ca4-35cd-41c5-a5fe-1e117217d095_calibrate_994765_51.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4bca1ca4-35cd-41c5-a5fe-1e117217d095_calibrate_994765_51.stderr 2024-03-15 04:14:55 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07404ad2920e0a26a3b12818b224f6f6 with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash a8ca9be470b7f398ada7fb1a796f38fa parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/2dbff8c4-972a-4698-8d7d-7e59b4cdbe18_calibrate_994766_13.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/2dbff8c4-972a-4698-8d7d-7e59b4cdbe18_calibrate_994766_13.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/fc07e084-d24c-4739-bb51-dc071799d400_characterizeImage_994766_13.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/fc07e084-d24c-4739-bb51-dc071799d400_characterizeImage_994766_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:56 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d35eda497a977d298632c110b4b220a with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c9d00fcf-5d7e-4c29-ae87-d878f19cb5c0_calibrate_994765_60.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c9d00fcf-5d7e-4c29-ae87-d878f19cb5c0_calibrate_994765_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:56 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dda74284a743c8a3433b9615b9699a9c with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash 79a3f9e5bf48eb2b7ee4ab5bc0bbcc75 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/fed133ab-4f1c-484d-a29f-a848098bde60_calibrate_994766_58.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/fed133ab-4f1c-484d-a29f-a848098bde60_calibrate_994766_58.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5c5420e7-7859-43df-9f20-0c69df7d1111_characterizeImage_994766_58.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/5c5420e7-7859-43df-9f20-0c69df7d1111_characterizeImage_994766_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:57 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5f22dada2479fa733d405ed015a230e with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash b9ed8d3815d2fd517ccd7d929116a396 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/75748dc1-7c90-4635-82a3-57ae3e6ab17d_calibrate_994765_50.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/75748dc1-7c90-4635-82a3-57ae3e6ab17d_calibrate_994765_50.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/03996248-ab55-46b5-85c1-694a689a9830_characterizeImage_994765_50.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/03996248-ab55-46b5-85c1-694a689a9830_characterizeImage_994765_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:57 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 943 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84b11041ff285128766aac6a6da50791 with result from task 943 parsl.dataflow.dflow DEBUG: Task 943 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 944 has memoization hash 613cb2def77d333dd0ae2345b80399b2 parsl.dataflow.memoization INFO: Task 944 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 944 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/6bb17817-00f5-4490-b28e-6a0a025acb1e_calibrate_994767_3.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/6bb17817-00f5-4490-b28e-6a0a025acb1e_calibrate_994767_3.stderr parsl.dataflow.dflow INFO: Standard output for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/bcc5439b-e38e-4559-ae8c-2eb23b6c6bab_characterizeImage_994767_3.stdout parsl.dataflow.dflow INFO: Standard error for task 943 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/bcc5439b-e38e-4559-ae8c-2eb23b6c6bab_characterizeImage_994767_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:58 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:58 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:59 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee3298ff03b786aaff061c9acda1a351 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 147d010124657f216f64447035f37a45 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f83f0258-a45c-4717-935e-69b083f187a6_calibrate_994766_50.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f83f0258-a45c-4717-935e-69b083f187a6_calibrate_994766_50.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/39b7ca8a-7104-48c3-a9db-32d081257021_characterizeImage_994766_50.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/39b7ca8a-7104-48c3-a9db-32d081257021_characterizeImage_994766_50.stderr parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b80e16b63a54d4470fff949e6687d20b with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash 5eca015f59dbc04e95c69563529625da parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63df100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/89e15f47-08f3-49b7-b581-f2d35ac4b6bf_calibrate_994765_1.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/89e15f47-08f3-49b7-b581-f2d35ac4b6bf_calibrate_994765_1.stderr parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c61016b9-fdfe-4a12-93d0-5aa66d1de94e_characterizeImage_994765_1.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c61016b9-fdfe-4a12-93d0-5aa66d1de94e_characterizeImage_994765_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:59 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5950b8110d6f76b5dc820ff4f145a9a with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7ea29db2-f0c7-4d73-8814-4ab8a5cce799_calibrate_994767_25.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7ea29db2-f0c7-4d73-8814-4ab8a5cce799_calibrate_994767_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:14:59 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:00 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:01 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16cc4e4d839a36eda348fe3ec1584365 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1229f817-55d4-435c-8b0b-3e20c5fbf081_calibrate_994767_60.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1229f817-55d4-435c-8b0b-3e20c5fbf081_calibrate_994767_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:01 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:01 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7989dc47a63ac8961a74d8ea23b833f5 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2ef91ce7-6b77-4412-9b63-9f09b65109ad_calibrate_994767_62.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2ef91ce7-6b77-4412-9b63-9f09b65109ad_calibrate_994767_62.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 258f73ce40d6d9de80135eb5daa818bf with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8a7f2cd2-7464-426e-a76b-0f285a0abdd0_calibrate_994767_51.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8a7f2cd2-7464-426e-a76b-0f285a0abdd0_calibrate_994767_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:02 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 991 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29e5e4cb3489d38cc496e9a656c09026 with result from task 991 parsl.dataflow.dflow DEBUG: Task 991 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 992 has memoization hash 74166d0d818d9f66bb11692f31598e3a parsl.dataflow.memoization INFO: Task 992 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 992 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/32e245af-ed45-4bb4-b762-94a3bcbc8fd0_calibrate_994765_47.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/32e245af-ed45-4bb4-b762-94a3bcbc8fd0_calibrate_994765_47.stderr parsl.dataflow.dflow INFO: Standard output for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8bf8052b-5d04-4aeb-89ae-9640eebcdf91_characterizeImage_994765_47.stdout parsl.dataflow.dflow INFO: Standard error for task 991 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/8bf8052b-5d04-4aeb-89ae-9640eebcdf91_characterizeImage_994765_47.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20b3a05066d3eff7851abdd8b1a2b167 with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f8bbc962-f5e0-434b-b268-5423a7ea6e20_calibrate_994765_10.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f8bbc962-f5e0-434b-b268-5423a7ea6e20_calibrate_994765_10.stderr parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c112b4d0164fe9d72f3178972c82826 with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 945e2905c7651112bfa92fd77115fa39 parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/cab0fdcf-9626-4332-945f-84c41489cfa7_calibrate_994765_33.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/cab0fdcf-9626-4332-945f-84c41489cfa7_calibrate_994765_33.stderr parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a3029440-f6c6-4cac-98ec-add63024ef96_characterizeImage_994765_33.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a3029440-f6c6-4cac-98ec-add63024ef96_characterizeImage_994765_33.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb48e2b3af5dde3b35913883d7a62fc4 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6e2d0e9b-d004-47cd-96c4-5e9ca894f387_calibrate_994766_26.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6e2d0e9b-d004-47cd-96c4-5e9ca894f387_calibrate_994766_26.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 593b6a9dc0e55da76699840b0936cc30 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/77c0460b-409e-4ff5-9849-0dbab3f1f680_calibrate_994766_10.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/77c0460b-409e-4ff5-9849-0dbab3f1f680_calibrate_994766_10.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b49386fb5d6a911655911a15e038105 with result from task 134 parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:03 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ea324901-ae37-4a76-8052-370a3aaa93ed_calibrate_994765_62.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ea324901-ae37-4a76-8052-370a3aaa93ed_calibrate_994765_62.stderr parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 931382552e23aa6a4f9d71712e13cd09 with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 7c9621fa31f7fcb956f7a48d889f36c5 parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/abe18b3b-ef0f-468b-bdc8-bbdb30b65c8e_calibrate_994765_59.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/abe18b3b-ef0f-468b-bdc8-bbdb30b65c8e_calibrate_994765_59.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ec8a1919-cb6a-44d5-93d3-dfdd1b44f592_characterizeImage_994765_59.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ec8a1919-cb6a-44d5-93d3-dfdd1b44f592_characterizeImage_994765_59.stderr parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 772c56ad436c9981c305f7eea2559dff with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash 63cfa7038618fe8aa43adf38c98bfc22 parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/72eaed7c-b588-4f2b-b546-3ae427f65a32_calibrate_994765_55.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/72eaed7c-b588-4f2b-b546-3ae427f65a32_calibrate_994765_55.stderr parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c329804d-19a9-4c53-8bfa-17ca299a1b36_characterizeImage_994765_55.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/c329804d-19a9-4c53-8bfa-17ca299a1b36_characterizeImage_994765_55.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f702432f988dabe042effb75d130484 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/6fd16fb6-9d59-4993-94cb-65bcee574ab0_calibrate_994765_31.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/6fd16fb6-9d59-4993-94cb-65bcee574ab0_calibrate_994765_31.stderr parsl.dataflow.dflow INFO: Task 976 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cb0035a16fbc51b3b69f09febb9e2eb with result from task 976 parsl.dataflow.dflow DEBUG: Task 976 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 977 has memoization hash 524b1e073f17415c0409c2999f8a3d57 parsl.dataflow.memoization INFO: Task 977 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 977 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/90a59d82-444e-4588-80b5-b7609a14ebe8_calibrate_994767_48.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/90a59d82-444e-4588-80b5-b7609a14ebe8_calibrate_994767_48.stderr parsl.dataflow.dflow INFO: Standard output for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/7df45191-73d4-49ac-9c7b-b19f8f4d2746_characterizeImage_994767_48.stdout parsl.dataflow.dflow INFO: Standard error for task 976 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/7df45191-73d4-49ac-9c7b-b19f8f4d2746_characterizeImage_994767_48.stderr parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81673456172120bda496382d437596b4 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/400873f3-88d4-4206-93e8-fef23c91fe5f_calibrate_994765_40.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/400873f3-88d4-4206-93e8-fef23c91fe5f_calibrate_994765_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:03 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 632e3157ed459c5fd694b7cc141c3877 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/78dfa728-5c5d-40dd-81e1-6d2d1697aff7_calibrate_994765_41.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/78dfa728-5c5d-40dd-81e1-6d2d1697aff7_calibrate_994765_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:03 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:04 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e8f5b2dae81312b7b108254ee8b0f63 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d7460846-a5b7-442a-9abf-caf9321e5f27_calibrate_994767_55.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d7460846-a5b7-442a-9abf-caf9321e5f27_calibrate_994767_55.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf8ade217ba9f8040a211893ec3a8f57 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ce06c7e0-2d6a-4174-8331-a2f8dfc26f8c_calibrate_994766_21.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ce06c7e0-2d6a-4174-8331-a2f8dfc26f8c_calibrate_994766_21.stderr parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64b23e5a868e741df5c3a627d77c05cd with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash f9c7e4a32ce238d113fb05a8fe0c5094 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b9e67ef3-c802-4342-94e6-1ea30e739630_calibrate_994766_6.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b9e67ef3-c802-4342-94e6-1ea30e739630_calibrate_994766_6.stderr parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/7b4214db-bbf2-4796-952a-7ba81e272228_characterizeImage_994766_6.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/7b4214db-bbf2-4796-952a-7ba81e272228_characterizeImage_994766_6.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c32c2b5c5480f06c3977b691b44620b4 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1267df00-9eac-45c3-a88d-bc3ff890ea43_calibrate_994767_31.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1267df00-9eac-45c3-a88d-bc3ff890ea43_calibrate_994767_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:04 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:05 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3caa0602b1a2e5f21dc9c288faf5527f with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/313b140c-3bf9-4fc9-962a-e967ab4de04d_calibrate_994765_6.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/313b140c-3bf9-4fc9-962a-e967ab4de04d_calibrate_994765_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:05 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72f5f729a9f3af8215b0182308c33273 with result from task 874 parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 12ee3bb1570f12ab04f34fc3efd69ae1 parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/e778cd77-a791-4268-a9dc-0a87279770cd_calibrate_994766_1.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/e778cd77-a791-4268-a9dc-0a87279770cd_calibrate_994766_1.stderr parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/617c516d-c7e2-46a6-bf1b-d32b42b96e33_characterizeImage_994766_1.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/617c516d-c7e2-46a6-bf1b-d32b42b96e33_characterizeImage_994766_1.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d74df2def58f5313fb8fe25db9dab55b with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c4923ac2-5777-49c6-bf5b-b1667d6060e6_calibrate_994765_34.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c4923ac2-5777-49c6-bf5b-b1667d6060e6_calibrate_994765_34.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad9a2bf53f652e82e8f767927e8f75cc with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash aa1d82de9084d4a1276d08d8a91b86a4 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/a5e9d841-a183-4e7d-9433-2a98a24fe62e_calibrate_994767_47.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/a5e9d841-a183-4e7d-9433-2a98a24fe62e_calibrate_994767_47.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/1bb53827-df77-4889-96b0-c1738278d664_characterizeImage_994767_47.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/1bb53827-df77-4889-96b0-c1738278d664_characterizeImage_994767_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:06 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffb766eff9e36c11c08825ef057ed4a6 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0a8a227a-8d92-45fd-b1d9-323b778e1ad5_calibrate_994766_52.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0a8a227a-8d92-45fd-b1d9-323b778e1ad5_calibrate_994766_52.stderr parsl.dataflow.dflow INFO: Task 965 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ccae2cfa65db9c291c1fb43230bc19b with result from task 965 parsl.dataflow.dflow DEBUG: Task 965 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/153e3e89-9c9b-4af5-bd69-9fc862b791e6_calibrate_994766_45.stdout parsl.dataflow.dflow INFO: Standard error for task 965 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/153e3e89-9c9b-4af5-bd69-9fc862b791e6_calibrate_994766_45.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56470ec47a15008e2e94b07fe2526737 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 8047247ca65bb2b3a9e7a005c93da1b6 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8b9f3b3-4772-422b-8652-f651d54704f0_calibrate_994766_31.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8b9f3b3-4772-422b-8652-f651d54704f0_calibrate_994766_31.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/6b170b20-9261-4338-b6e7-d8f857536d87_characterizeImage_994766_31.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/6b170b20-9261-4338-b6e7-d8f857536d87_characterizeImage_994766_31.stderr parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f8f608a20ba4933ed4ff0efd679847c with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/8afac479-8583-4bbc-98f0-8a4ebd7a8f36_calibrate_994765_18.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/8afac479-8583-4bbc-98f0-8a4ebd7a8f36_calibrate_994765_18.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c6ade5fefefe2d2164fbfb18620ae16 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/5aea1c6e-53ff-4b41-b051-34a7a85e90e1_calibrate_994765_27.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/5aea1c6e-53ff-4b41-b051-34a7a85e90e1_calibrate_994765_27.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f4ce6dc4caf34ed3afcc74b372ae173 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f453bd8f-58e9-471a-8e6e-90ab8f74554d_calibrate_994766_7.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f453bd8f-58e9-471a-8e6e-90ab8f74554d_calibrate_994766_7.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a26daee3b3309ab6c0b02d215ae69155 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/7ea8637e-a7a3-43a5-9300-4dd0ebd3c1ee_calibrate_994765_25.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/7ea8637e-a7a3-43a5-9300-4dd0ebd3c1ee_calibrate_994765_25.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18708103614cdef0c160b46dbd3fa3b8 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c1be9e28-f229-4575-af6a-0ae01206ff97_calibrate_994767_24.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c1be9e28-f229-4575-af6a-0ae01206ff97_calibrate_994767_24.stderr parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ea8bd27321f39efd05d6d8b0952e641 with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f6df9893-416b-41dc-8477-614b41793f1e_calibrate_994765_48.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f6df9893-416b-41dc-8477-614b41793f1e_calibrate_994765_48.stderr parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ad2b642445f24e852e11d5215739272 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/53f6f463-1b36-4bc7-a167-172b6f6e852a_calibrate_994765_7.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/53f6f463-1b36-4bc7-a167-172b6f6e852a_calibrate_994765_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:08 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28defee3178d6a1c2c3dca548fe67ca8 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0bcc6c94-9777-47ac-8b92-51227ba50ad3_calibrate_994766_59.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0bcc6c94-9777-47ac-8b92-51227ba50ad3_calibrate_994766_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:08 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fd0e69db55a12c3082a374769c64c7d with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 537457696c8ec3bd42c7507b6b45916c parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66000e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/eb2ddf91-6913-4db4-a2a6-de72f71337a3_calibrate_994639_16.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/eb2ddf91-6913-4db4-a2a6-de72f71337a3_calibrate_994639_16.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/347962fb-3d86-491d-b465-43139a4289dc_characterizeImage_994639_16.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/347962fb-3d86-491d-b465-43139a4289dc_characterizeImage_994639_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:08 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8ef07bdf86a390797152fd095c53e38 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4a468632-ae10-48f7-ab12-06aafca0130d_calibrate_994765_57.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4a468632-ae10-48f7-ab12-06aafca0130d_calibrate_994765_57.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5009df8164c5fca88cb294cea66466aa with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/15856891-ae06-445f-99a6-a502e1fe7bf5_calibrate_994766_25.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/15856891-ae06-445f-99a6-a502e1fe7bf5_calibrate_994766_25.stderr parsl.dataflow.dflow INFO: Task 1097 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f92955dd37af412259a6c9b853e5e9a with result from task 1097 parsl.dataflow.dflow DEBUG: Task 1097 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/60c98210-d613-42c9-826e-f7eb90fd0d10_calibrate_994765_23.stdout parsl.dataflow.dflow INFO: Standard error for task 1097 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/60c98210-d613-42c9-826e-f7eb90fd0d10_calibrate_994765_23.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d1e8cd34c466b29d174ff178e2d9ad7 with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 9c02609c1df97aed783da43b236f751f parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66505e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/05bb53a7-c9bf-4318-b11b-d3e1bf4c1e0b_calibrate_994767_43.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/05bb53a7-c9bf-4318-b11b-d3e1bf4c1e0b_calibrate_994767_43.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/241fbe34-1149-4d9c-ae4f-473dddd7a7f8_characterizeImage_994767_43.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/241fbe34-1149-4d9c-ae4f-473dddd7a7f8_characterizeImage_994767_43.stderr parsl.dataflow.dflow INFO: Task 733 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c7b3b80289e4ecc688dd80e42835d0d with result from task 733 parsl.dataflow.dflow DEBUG: Task 733 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 734 has memoization hash 56a1fd27e5c996a4f2426f0c2931a109 parsl.dataflow.memoization INFO: Task 734 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 734 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/957e97d9-10da-40f9-b604-be50cf4802d9_calibrate_994765_3.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/957e97d9-10da-40f9-b604-be50cf4802d9_calibrate_994765_3.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/620f7ba6-aca5-4121-8486-275d90b43ced_characterizeImage_994765_3.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/620f7ba6-aca5-4121-8486-275d90b43ced_characterizeImage_994765_3.stderr parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ceadf86c773a2804d734defd933a76e6 with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 804bbe7adefb978db055da4d3a273fca parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63913a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/bc8261bc-a02a-49f0-8099-e22c54d359b7_calibrate_994765_26.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/bc8261bc-a02a-49f0-8099-e22c54d359b7_calibrate_994765_26.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/965702f0-5260-4305-b6ba-22be2e3be93e_characterizeImage_994765_26.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/965702f0-5260-4305-b6ba-22be2e3be93e_characterizeImage_994765_26.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2cb84a1fa1880f7c23fa37a75a7e173 with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8b5ad807-af06-4631-9519-3c920d523d7c_calibrate_994766_39.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8b5ad807-af06-4631-9519-3c920d523d7c_calibrate_994766_39.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6cedda2b9df0a9e02d94359a832272ec with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash d59113fd924ae45775da4c23df1a4631 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a76a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/bc5219a4-12bd-4b46-8111-dc6f45a62f1f_calibrate_994765_5.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/bc5219a4-12bd-4b46-8111-dc6f45a62f1f_calibrate_994765_5.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/eebf294a-f6a1-4307-9de0-f95892bc4ae3_characterizeImage_994765_5.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/eebf294a-f6a1-4307-9de0-f95892bc4ae3_characterizeImage_994765_5.stderr parsl.dataflow.dflow INFO: Task 949 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14471c8893d7903477b31ffc9999fc95 with result from task 949 parsl.dataflow.dflow DEBUG: Task 949 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 950 has memoization hash b7b56eefee58175525f65d89faf94387 parsl.dataflow.memoization INFO: Task 950 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 950 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/a3400530-229e-46ac-87c8-4bd8db12d5a5_calibrate_994767_59.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/a3400530-229e-46ac-87c8-4bd8db12d5a5_calibrate_994767_59.stderr parsl.dataflow.dflow INFO: Standard output for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/f18301df-8b67-47c7-93db-7116f6f7f587_characterizeImage_994767_59.stdout parsl.dataflow.dflow INFO: Standard error for task 949 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/f18301df-8b67-47c7-93db-7116f6f7f587_characterizeImage_994767_59.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92e64f070709aaf4e2f71c3a6394333d with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash bd6ed9a160d5762d8f14d1fddd982185 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/364f84d5-570f-4398-849a-44bee4065bc5_calibrate_994766_49.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/364f84d5-570f-4398-849a-44bee4065bc5_calibrate_994766_49.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c605c894-b887-4ee3-b08a-7f86c0569b8d_characterizeImage_994766_49.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/c605c894-b887-4ee3-b08a-7f86c0569b8d_characterizeImage_994766_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:10 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:10 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d54aa956edc3fa2b7928b8668a8ebc72 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 6bb4e1a48d1125f9debe1d51bb63bf37 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b1b7f472-e155-4de9-9cde-ba507bf3792c_calibrate_994767_10.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b1b7f472-e155-4de9-9cde-ba507bf3792c_calibrate_994767_10.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/89a01609-1000-493d-8db2-6c70d7a90970_characterizeImage_994767_10.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/89a01609-1000-493d-8db2-6c70d7a90970_characterizeImage_994767_10.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc4c6eee58a80ee9f079cab125e41061 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f58a330c-e3b7-46bf-9026-774d718b0c6a_calibrate_994766_23.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f58a330c-e3b7-46bf-9026-774d718b0c6a_calibrate_994766_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:10 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:10 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7c302d0dcfc8b3dff5e00c9228d182a with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/281b0927-cce3-4063-8c65-31170fd7ded5_calibrate_994767_57.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/281b0927-cce3-4063-8c65-31170fd7ded5_calibrate_994767_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:10 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64f06a13f44d27ad9e69ac15979808e3 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6f13856b-adc8-4793-b0a0-70dcf742e270_calibrate_994766_16.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6f13856b-adc8-4793-b0a0-70dcf742e270_calibrate_994766_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:10 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1374540f4975b7f6db52205d93f5b786 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/e0d69843-dc27-4d33-9eee-569fee7625d8_calibrate_994767_21.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/e0d69843-dc27-4d33-9eee-569fee7625d8_calibrate_994767_21.stderr parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ebe7fae65c84f5876d17ddea4436064 with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/76b134bb-25d8-40c5-aa3b-7f07125d2af3_calibrate_994765_22.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/76b134bb-25d8-40c5-aa3b-7f07125d2af3_calibrate_994765_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:10 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22f482dd6683ad5e356b94ad43befdc6 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/58c69f05-94a6-4ac0-a4fa-159d5016141f_calibrate_994767_11.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/58c69f05-94a6-4ac0-a4fa-159d5016141f_calibrate_994767_11.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f00d469ccf33caa0aebc950a2d008de8 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ed29c0ea-ba96-4fb1-a767-016a3df73cc6_calibrate_994766_19.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ed29c0ea-ba96-4fb1-a767-016a3df73cc6_calibrate_994766_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:11 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 088597bbdc3e6a4520f72fdc2d8e4f4b with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/7bf3ac5f-2e3a-4f7e-85a8-c7743b313bb4_calibrate_994766_36.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/7bf3ac5f-2e3a-4f7e-85a8-c7743b313bb4_calibrate_994766_36.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bffd49c32dc79d8662c5cf6c309af930 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash a9711e31ff8bf1eedb67b16d41283007 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b50c7c5e-bf41-4fc8-bc64-c2dc7221852d_calibrate_994766_37.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b50c7c5e-bf41-4fc8-bc64-c2dc7221852d_calibrate_994766_37.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/d617abdf-0be1-4635-a57b-f0e822b8807d_characterizeImage_994766_37.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/d617abdf-0be1-4635-a57b-f0e822b8807d_characterizeImage_994766_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:11 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8e17535c4bc1f6a88e37aaa40eca3a6 with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash 31e9393643ac51882d80c7bfd5a22600 parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/fa9a8386-a0ca-4cca-85af-b4800c8cdd03_calibrate_994765_36.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/fa9a8386-a0ca-4cca-85af-b4800c8cdd03_calibrate_994765_36.stderr parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/e9dfa96b-a3f6-4380-a314-2948cda9dc18_characterizeImage_994765_36.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/e9dfa96b-a3f6-4380-a314-2948cda9dc18_characterizeImage_994765_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:11 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 1016 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2feacbcb09bada67a8f2637f0a3df198 with result from task 1016 parsl.dataflow.dflow DEBUG: Task 1016 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/e19bbdc3-99b5-460c-8e2d-878504a44744_calibrate_994767_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1016 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/e19bbdc3-99b5-460c-8e2d-878504a44744_calibrate_994767_56.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7611d61860aab6406519eca932ae96c5 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/447f2fee-af9a-49a0-9663-63d9e9235dfc_calibrate_994766_60.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/447f2fee-af9a-49a0-9663-63d9e9235dfc_calibrate_994766_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:11 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:11 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f709a79e6293d34f298625fcf8169b1 with result from task 202 parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:12 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash c3bc3d5cfb6922f4059f21804daf04ae parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/a86a9f4a-a701-422c-b2c5-30368c51b8c2_calibrate_994766_43.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/a86a9f4a-a701-422c-b2c5-30368c51b8c2_calibrate_994766_43.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/90809a26-b2bc-46e1-85b9-d8881f1f7c89_characterizeImage_994766_43.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/90809a26-b2bc-46e1-85b9-d8881f1f7c89_characterizeImage_994766_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:12 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0303759a025398b144983105ffe3dd6c with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2a62b54a-b4ea-4240-8a7a-982660cf3dbb_calibrate_994765_52.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2a62b54a-b4ea-4240-8a7a-982660cf3dbb_calibrate_994765_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:12 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:12 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:12 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:12 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 1027 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e75357728cb20e8f3e835ef006edff0 with result from task 1027 parsl.dataflow.dflow DEBUG: Task 1027 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1028 has memoization hash 7383d9df057450d469c28ade377aa604 parsl.dataflow.memoization INFO: Task 1028 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ec540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1028 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/96d94869-046e-421d-a6e9-c3d5f2b15042_calibrate_994765_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/96d94869-046e-421d-a6e9-c3d5f2b15042_calibrate_994765_8.stderr parsl.dataflow.dflow INFO: Standard output for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b74f82b3-e449-431c-a08e-f4e9949e363d_characterizeImage_994765_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1027 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/b74f82b3-e449-431c-a08e-f4e9949e363d_characterizeImage_994765_8.stderr parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cbc06e403e56462450575b0c3b58144 with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash 3bbc489f90e963598175f14b5c2669c4 parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/553b401a-2a99-4d7c-82b6-d0d7553b49cf_calibrate_994765_42.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/553b401a-2a99-4d7c-82b6-d0d7553b49cf_calibrate_994765_42.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/beee8377-dc6d-460c-883f-5fff58d46207_characterizeImage_994765_42.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/beee8377-dc6d-460c-883f-5fff58d46207_characterizeImage_994765_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:12 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44be1a929d38b9fd62ff433c0f7b4e17 with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2656270a-e7a7-4e40-bd83-2bd0b3676570_calibrate_994767_9.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2656270a-e7a7-4e40-bd83-2bd0b3676570_calibrate_994767_9.stderr parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de308c0b33535c4a92b28112384ace90 with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/9e5c93e2-0731-4793-a72d-d01786d43999_calibrate_994766_40.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/9e5c93e2-0731-4793-a72d-d01786d43999_calibrate_994766_40.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59bcf7d41ae8386c69a8620e3e7f20cd with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0aa80de6-af7d-4bc6-9ec2-85da45f5ff03_calibrate_994766_18.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/0aa80de6-af7d-4bc6-9ec2-85da45f5ff03_calibrate_994766_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:13 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57b327d34d0383b5ffd5ce43f8838b9f with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f2b7962a-0ef6-4709-a013-be0c847912ee_calibrate_994765_39.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f2b7962a-0ef6-4709-a013-be0c847912ee_calibrate_994765_39.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cdaf75c7f7cff6c60ab22ee2738f5d0 with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 9a0e2c6084011e89072b78a198e184ad parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/39db03f4-e8a3-41fe-bde0-5d6b7021900e_calibrate_994639_31.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/39db03f4-e8a3-41fe-bde0-5d6b7021900e_calibrate_994639_31.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/919854bf-d625-46cd-93f9-7412933f08be_characterizeImage_994639_31.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/919854bf-d625-46cd-93f9-7412933f08be_characterizeImage_994639_31.stderr parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 412189598951d684dfd68960263bd0f0 with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash 38a3d062afba5c24d056f1b707e65e60 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66534c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b6376663-80fe-4317-a2bf-92d50b2ff5eb_calibrate_994765_9.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b6376663-80fe-4317-a2bf-92d50b2ff5eb_calibrate_994765_9.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/0abeab3f-d2ec-40b2-9f40-02eecc84c771_characterizeImage_994765_9.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/0abeab3f-d2ec-40b2-9f40-02eecc84c771_characterizeImage_994765_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:13 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ada352ecc7a42daedd10a5bde4877624 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/297c5151-dbc0-4d53-aace-e061bc6bc8b6_calibrate_994767_18.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/297c5151-dbc0-4d53-aace-e061bc6bc8b6_calibrate_994767_18.stderr parsl.dataflow.dflow INFO: Task 1021 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2671b47ba9a052840431c9154f8f6cef with result from task 1021 parsl.dataflow.dflow DEBUG: Task 1021 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1022 has memoization hash 61e0162075c0feef24e510148d3e4c07 parsl.dataflow.memoization INFO: Task 1022 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1022 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2de7bf58-16d9-46b1-adc6-4988d4d9d743_calibrate_994765_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2de7bf58-16d9-46b1-adc6-4988d4d9d743_calibrate_994765_17.stderr parsl.dataflow.dflow INFO: Standard output for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/399442c0-995f-4f80-890e-3383d6c0ea8d_characterizeImage_994765_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1021 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/399442c0-995f-4f80-890e-3383d6c0ea8d_characterizeImage_994765_17.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffb40751c3b6179f9744812ce7cf10ec with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/788e83af-d439-401d-8898-8ef14a9eaac0_calibrate_994767_14.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/788e83af-d439-401d-8898-8ef14a9eaac0_calibrate_994767_14.stderr parsl.dataflow.dflow INFO: Task 1039 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9827304be4177fbe7d3e7819d323c13f with result from task 1039 parsl.dataflow.dflow DEBUG: Task 1039 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1040 has memoization hash 7c6b679c18997615d71cdcabde14622f parsl.dataflow.memoization INFO: Task 1040 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ed300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1040 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2b87c2f1-0cad-4d62-b4d4-ade1cfc37a21_calibrate_994765_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2b87c2f1-0cad-4d62-b4d4-ade1cfc37a21_calibrate_994765_29.stderr parsl.dataflow.dflow INFO: Standard output for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ed7a8b1b-5157-4f6c-92f8-b1dda9bf3ff7_characterizeImage_994765_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1039 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/ed7a8b1b-5157-4f6c-92f8-b1dda9bf3ff7_characterizeImage_994765_29.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ec843f393c9df1b0b9fa5c984fb1b11 with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/e710dd07-232d-45cd-a7e7-288d7e419fc3_calibrate_994765_19.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/e710dd07-232d-45cd-a7e7-288d7e419fc3_calibrate_994765_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:14 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 956 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 260d7494fa2b0af5a9629f1bc03bf2a6 with result from task 956 parsl.dataflow.dflow DEBUG: Task 956 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/054d1ec5-f725-4401-a354-1a34e12c26c4_calibrate_994765_21.stdout parsl.dataflow.dflow INFO: Standard error for task 956 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/054d1ec5-f725-4401-a354-1a34e12c26c4_calibrate_994765_21.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff68618ed33b0d817ded9fdb1d115dd7 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/fdccfb17-6578-4621-9e65-23f4cf8aa8a6_calibrate_994767_53.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/fdccfb17-6578-4621-9e65-23f4cf8aa8a6_calibrate_994767_53.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97d6abaa9412124f35d4119604627150 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash eba8b1b5e85c57e7a677ac833b69da4b parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b87380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/451ef0ea-d28b-406c-84cc-341479b835ed_calibrate_994640_46.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/451ef0ea-d28b-406c-84cc-341479b835ed_calibrate_994640_46.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8a19faaf-da84-4111-8019-b590d28fbcc8_characterizeImage_994640_46.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8a19faaf-da84-4111-8019-b590d28fbcc8_characterizeImage_994640_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:15 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06749810c92d60c62ada5554d6c097dc with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 86792a61ea73a31ccf6d279d49155f9a parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6d504038-0779-4ef8-8070-1130239edbaf_calibrate_994766_47.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6d504038-0779-4ef8-8070-1130239edbaf_calibrate_994766_47.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/7387aa2c-5f3e-452c-8877-b8c5876c4c42_characterizeImage_994766_47.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/7387aa2c-5f3e-452c-8877-b8c5876c4c42_characterizeImage_994766_47.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91d7f370edf2ce24c6183f298c3c45a3 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash cf1990d22ac524500aafad2ae6f25623 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6600a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/87eec224-9981-44ec-aba7-e935df284ad5_calibrate_994639_39.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/87eec224-9981-44ec-aba7-e935df284ad5_calibrate_994639_39.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c700767c-5a5e-44f1-aa42-bf9e054cdf87_characterizeImage_994639_39.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c700767c-5a5e-44f1-aa42-bf9e054cdf87_characterizeImage_994639_39.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d5fddca7ca0de905eb576f7a705717c with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 47a72409193e1efa5d8b34de831afff3 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e5800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/9866727d-abe2-4db3-a961-2bbcfcf2e9dd_calibrate_994640_4.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/9866727d-abe2-4db3-a961-2bbcfcf2e9dd_calibrate_994640_4.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/77861902-727d-43fa-b1ea-defa7adc7cf5_characterizeImage_994640_4.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/77861902-727d-43fa-b1ea-defa7adc7cf5_characterizeImage_994640_4.stderr parsl.dataflow.dflow INFO: Task 1042 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e061f084006a1a8b9798e40b18cf54f5 with result from task 1042 parsl.dataflow.dflow DEBUG: Task 1042 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1043 has memoization hash 315a6c86b823b74f7d3d587f812d31cb parsl.dataflow.memoization INFO: Task 1043 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ed620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1043 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/770a5a5b-0d70-496c-9613-35617df475de_calibrate_994765_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/770a5a5b-0d70-496c-9613-35617df475de_calibrate_994765_13.stderr parsl.dataflow.dflow INFO: Standard output for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f563be75-e0d3-47fe-9a96-a09f2c9b4210_characterizeImage_994765_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1042 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f563be75-e0d3-47fe-9a96-a09f2c9b4210_characterizeImage_994765_13.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe6a198b81ff6c561369a28b67192b0b with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash dde79f9526e970abad88de02da4e185d parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b876a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/90b566e7-895d-4b29-b4b9-1662ae18885b_calibrate_994638_15.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/90b566e7-895d-4b29-b4b9-1662ae18885b_calibrate_994638_15.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b6f3a249-32ed-4c0c-8c2f-d30902698093_characterizeImage_994638_15.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b6f3a249-32ed-4c0c-8c2f-d30902698093_characterizeImage_994638_15.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77c22736dd57c8071b2a8c5f8462669b with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8f974d6f-da2e-4d52-85bf-6e64430d7d02_calibrate_994767_30.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8f974d6f-da2e-4d52-85bf-6e64430d7d02_calibrate_994767_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:15 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 997ea847b23e772cdcdb98609a0b1ac3 with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 532bccfbe55b05cedf13535c72a03d4e parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b8762ba7-8756-40bc-9538-6c4fa950e1a9_calibrate_994640_15.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b8762ba7-8756-40bc-9538-6c4fa950e1a9_calibrate_994640_15.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e87f76d3-1358-4099-9a05-d3f84cea725e_characterizeImage_994640_15.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e87f76d3-1358-4099-9a05-d3f84cea725e_characterizeImage_994640_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:15 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9d18afef13889413e6934b547d684d2 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash c44970e1dd0737db1e0045be01ab3645 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e51c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/302dff4d-8fae-474a-bcba-685f464bc677_calibrate_994639_24.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/302dff4d-8fae-474a-bcba-685f464bc677_calibrate_994639_24.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7c399678-9753-4a78-b508-09d4f5ac42c5_characterizeImage_994639_24.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7c399678-9753-4a78-b508-09d4f5ac42c5_characterizeImage_994639_24.stderr parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bb84e0f112c6249b3809720ffe387d3 with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash 47da7e715b8293f452a590eeeca3e150 parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f2b02d47-7da2-4a95-8c98-5711f02e07e0_calibrate_994765_24.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f2b02d47-7da2-4a95-8c98-5711f02e07e0_calibrate_994765_24.stderr parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/5d992288-6fa2-4cd4-9336-7cc4fc887bcc_characterizeImage_994765_24.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/5d992288-6fa2-4cd4-9336-7cc4fc887bcc_characterizeImage_994765_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:16 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c624f13885eff196402bbb5044271756 with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash 048a5f4fe9938015685c5b0bc88b759d parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2213984e-d5fb-44cc-b8b4-0eacbfa73acd_calibrate_994767_58.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2213984e-d5fb-44cc-b8b4-0eacbfa73acd_calibrate_994767_58.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/dcfaf225-f135-4b67-85f3-ed6cfcab55d8_characterizeImage_994767_58.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/dcfaf225-f135-4b67-85f3-ed6cfcab55d8_characterizeImage_994767_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:16 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3caaa08dca41c5c950caad5a13e466b3 with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/206c150f-99ae-43f6-acb0-ecd11f8fd61c_calibrate_994765_44.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/206c150f-99ae-43f6-acb0-ecd11f8fd61c_calibrate_994765_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:17 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94354bc7d8a8ff82c6bf4acf813c2d9f with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c7192578-4fcd-45b2-95e9-9aaa8be648d6_calibrate_994767_20.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c7192578-4fcd-45b2-95e9-9aaa8be648d6_calibrate_994767_20.stderr parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e17fad86c7908f2f3436418f1a99dde with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash cea7f8b8773e06a7ed476ce455538645 parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8bd78f96-89bc-4997-98cc-be7b3c269b26_calibrate_994767_50.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8bd78f96-89bc-4997-98cc-be7b3c269b26_calibrate_994767_50.stderr parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/b194cd70-ddd2-44c0-ae78-e8029303331c_characterizeImage_994767_50.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/b194cd70-ddd2-44c0-ae78-e8029303331c_characterizeImage_994767_50.stderr parsl.dataflow.dflow INFO: Task 1108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4e2b69dfe4040112ebdd4b0ac8f0728 with result from task 1108 parsl.dataflow.dflow DEBUG: Task 1108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1109 has memoization hash 8480fd25e6cba92325da469501885ac6 parsl.dataflow.memoization INFO: Task 1109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1109 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3dee29e7-9d89-4709-a404-5c86befcf59d_calibrate_994767_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3dee29e7-9d89-4709-a404-5c86befcf59d_calibrate_994767_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/62b0291d-ca93-43d9-a8a5-173c8f238cfd_characterizeImage_994767_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/62b0291d-ca93-43d9-a8a5-173c8f238cfd_characterizeImage_994767_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:17 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 974 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e3b877e97e657b9618045d8d2b1c010 with result from task 974 parsl.dataflow.dflow DEBUG: Task 974 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b1744763-105b-4162-9fe3-4f2777beb982_calibrate_994767_6.stdout parsl.dataflow.dflow INFO: Standard error for task 974 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b1744763-105b-4162-9fe3-4f2777beb982_calibrate_994767_6.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc99bc368f9b92ca6c9b7c1673bc80a7 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ee3215b7-5903-4f1b-a03f-eaa6bdba0bb9_calibrate_994767_27.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ee3215b7-5903-4f1b-a03f-eaa6bdba0bb9_calibrate_994767_27.stderr parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a64085943f68ce10d74da0ce3035640 with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/bbff916a-2c69-44dd-8320-5ea54974bb8f_calibrate_994767_5.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/bbff916a-2c69-44dd-8320-5ea54974bb8f_calibrate_994767_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:17 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcec12c61e2067cd70170e9dcfd14052 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/073d30e0-4ad6-4a1d-b121-0e313156018f_calibrate_994765_20.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/073d30e0-4ad6-4a1d-b121-0e313156018f_calibrate_994765_20.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 088afe3ff33d316003d99231ef3ccaf7 with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/33c7a2ec-1ec6-478f-9d6e-81efa6be96e0_calibrate_994767_54.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/33c7a2ec-1ec6-478f-9d6e-81efa6be96e0_calibrate_994767_54.stderr parsl.dataflow.dflow INFO: Task 1069 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44f8cac5fca181cc3e0e53e474fa1da9 with result from task 1069 parsl.dataflow.dflow DEBUG: Task 1069 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1070 has memoization hash 79062b00a9b63d3c16cf6cbf28be3505 parsl.dataflow.memoization INFO: Task 1070 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62eef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1070 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7202fa5a-d952-4af7-a436-86a519c91202_calibrate_994767_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7202fa5a-d952-4af7-a436-86a519c91202_calibrate_994767_44.stderr parsl.dataflow.dflow INFO: Standard output for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/acfbc825-408f-4902-a04d-a3ab352d8b34_characterizeImage_994767_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1069 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/acfbc825-408f-4902-a04d-a3ab352d8b34_characterizeImage_994767_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:18 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2450fb3dcf86de8382d398c5a6e2552c with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8d99af81-c39e-4683-8c95-91fdef59c5fb_calibrate_994767_28.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8d99af81-c39e-4683-8c95-91fdef59c5fb_calibrate_994767_28.stderr parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70d716c3aab54bf141a34f6e30b3b41f with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash de1f3c9f75561e18cad5d3b9ec535b6f parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/06131462-475d-41db-993d-0e9b04908e72_calibrate_994765_4.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/06131462-475d-41db-993d-0e9b04908e72_calibrate_994765_4.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a3c3865f-59b6-4afc-bb76-376839ac3d08_characterizeImage_994765_4.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/a3c3865f-59b6-4afc-bb76-376839ac3d08_characterizeImage_994765_4.stderr parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e63ca1ddd6ded40367d8b7d5c3fcac7f with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/d8ebb432-b730-4eb3-93ae-97e85eca07ac_calibrate_994766_22.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/d8ebb432-b730-4eb3-93ae-97e85eca07ac_calibrate_994766_22.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 126dd9186653dfc162c1f46a16167d67 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/112b98c6-ac16-4a59-ad3e-9fa08038c4cc_calibrate_994767_34.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/112b98c6-ac16-4a59-ad3e-9fa08038c4cc_calibrate_994767_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:18 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:18 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d57ec6a8e30f06ce81dbbdf797f058c with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash 29460cb733d572c5e37662f3e6e93c93 parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62680e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/fd616599-937b-479a-9854-d8a9b6bddf21_calibrate_994765_58.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/fd616599-937b-479a-9854-d8a9b6bddf21_calibrate_994765_58.stderr parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/2be6d322-6f97-4d02-a91a-8f587a219b64_characterizeImage_994765_58.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/2be6d322-6f97-4d02-a91a-8f587a219b64_characterizeImage_994765_58.stderr parsl.dataflow.dflow INFO: Task 1024 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd541f1733090186ecf03bd53f7597b8 with result from task 1024 parsl.dataflow.dflow DEBUG: Task 1024 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1025 has memoization hash 941797b4ef6e1b6fbdbdc8d8d5ef2dc4 parsl.dataflow.memoization INFO: Task 1025 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1025 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/455f5211-b7e1-41c9-b893-ab0b54fb328e_calibrate_994766_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/455f5211-b7e1-41c9-b893-ab0b54fb328e_calibrate_994766_4.stderr parsl.dataflow.dflow INFO: Standard output for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/91af87b8-b39a-4672-9321-582034b92b93_characterizeImage_994766_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1024 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/91af87b8-b39a-4672-9321-582034b92b93_characterizeImage_994766_4.stderr parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ebfe3135fb76bcd23f56da8a4628d29 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ff536819-e375-4b4a-ba95-30b14eb371a0_calibrate_994766_56.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ff536819-e375-4b4a-ba95-30b14eb371a0_calibrate_994766_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:18 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:19 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 967c176ebf751f39709b2346c01de317 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 571d58c876b640f277c2efdb00a5f498 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a4e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/c39e77b1-541c-43b1-b451-f779177a7165_calibrate_994766_30.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/c39e77b1-541c-43b1-b451-f779177a7165_calibrate_994766_30.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e3acbf1c-9646-440c-8c9b-daf90e306dc0_characterizeImage_994766_30.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e3acbf1c-9646-440c-8c9b-daf90e306dc0_characterizeImage_994766_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:19 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d4988c2172ec7cf6fc1b3ba86887df6 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash bd56c4e1ddc5546625e98769d39f7640 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/630770c5-d86f-4243-bcf2-43ea926835c6_calibrate_994639_6.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/630770c5-d86f-4243-bcf2-43ea926835c6_calibrate_994639_6.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/80ef293c-15f0-4a59-be36-8fa3f2ea831e_characterizeImage_994639_6.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/80ef293c-15f0-4a59-be36-8fa3f2ea831e_characterizeImage_994639_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:19 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85ff2836e7c24b59231dbebe950dc350 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a8a7fdd1-f8e1-43d7-a4ce-e1cae5825b7b_calibrate_994765_38.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a8a7fdd1-f8e1-43d7-a4ce-e1cae5825b7b_calibrate_994765_38.stderr parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71daa22c9fd812354ba2f1d1ce577480 with result from task 751 parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash c12316fffd7e4bf3b99365c8592ad247 parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6392c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/ad519186-57a3-4cf4-aba0-fb95ae8d9281_calibrate_994640_33.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/ad519186-57a3-4cf4-aba0-fb95ae8d9281_calibrate_994640_33.stderr parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b93f87ee-ba60-4cf7-8830-8ec24c59d345_characterizeImage_994640_33.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b93f87ee-ba60-4cf7-8830-8ec24c59d345_characterizeImage_994640_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:19 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1048 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48da6e15b8e00add84841a09094fbd15 with result from task 1048 parsl.dataflow.dflow DEBUG: Task 1048 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1049 has memoization hash 8297ba6052151ea834602c11eb3c3071 parsl.dataflow.memoization INFO: Task 1049 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62edc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1049 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c01279f1-df36-4d2c-9120-4bb8b5742ac1_calibrate_994765_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c01279f1-df36-4d2c-9120-4bb8b5742ac1_calibrate_994765_56.stderr parsl.dataflow.dflow INFO: Standard output for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/e7646def-0cea-4061-b622-78d92b6fe98b_characterizeImage_994765_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1048 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/e7646def-0cea-4061-b622-78d92b6fe98b_characterizeImage_994765_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:19 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:19 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efe8c9a3609cacee4dc157859902cdc2 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 1f54ecf7505c2bbb769d3f1b8731f665 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66519e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/af08614a-4ba9-419d-b53c-fd969fc981e6_calibrate_994639_26.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/af08614a-4ba9-419d-b53c-fd969fc981e6_calibrate_994639_26.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c66dbeca-c198-48fd-b04a-bf6a48f944cf_characterizeImage_994639_26.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c66dbeca-c198-48fd-b04a-bf6a48f944cf_characterizeImage_994639_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:20 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b46a067b509f8834db2c3ed0bcd2449 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b76bba7d-4ba6-45d5-bc89-996f6b7272b0_calibrate_994767_37.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b76bba7d-4ba6-45d5-bc89-996f6b7272b0_calibrate_994767_37.stderr parsl.dataflow.dflow INFO: Task 1030 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c1c1011a108ebd128c91fde30a0b086 with result from task 1030 parsl.dataflow.dflow DEBUG: Task 1030 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1031 has memoization hash e6ac8ec1416a4742dac16ecb8ce3df8c parsl.dataflow.memoization INFO: Task 1031 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ec9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1031 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/db4104bc-36fe-4224-8ee3-7c410e772ac6_calibrate_994766_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/db4104bc-36fe-4224-8ee3-7c410e772ac6_calibrate_994766_46.stderr parsl.dataflow.dflow INFO: Standard output for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/abc325be-94d4-4d13-8abe-d8ab413cd10d_characterizeImage_994766_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1030 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/abc325be-94d4-4d13-8abe-d8ab413cd10d_characterizeImage_994766_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:20 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:20 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e65e63a90c24b26b1456953ebd417059 with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/d4279bbc-b62a-4d9d-85ec-f22e3d48f69c_calibrate_994766_32.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/d4279bbc-b62a-4d9d-85ec-f22e3d48f69c_calibrate_994766_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:20 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9feace7141b3550d662da29d2cf063a with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash 1d45ce1eff3e96e5ad717322ad9919e3 parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63df2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8242be81-bb94-45a8-a6dc-c54a027dcf68_calibrate_994766_34.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8242be81-bb94-45a8-a6dc-c54a027dcf68_calibrate_994766_34.stderr parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2b31f3d6-b0bc-40ed-98fa-d7c94fd38e29_characterizeImage_994766_34.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2b31f3d6-b0bc-40ed-98fa-d7c94fd38e29_characterizeImage_994766_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:20 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e29febd9bdc30adee9c0374f12610b6 with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ea9685bd-a71b-4114-a686-a24e46643823_calibrate_994766_62.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ea9685bd-a71b-4114-a686-a24e46643823_calibrate_994766_62.stderr parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92c5a63f21e57ecb864f6a216afdc259 with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/1e4f7f36-047d-491c-95b2-feed56294ca7_calibrate_994766_14.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/1e4f7f36-047d-491c-95b2-feed56294ca7_calibrate_994766_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:21 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42555677bb39d8b7469db8d9d89f9664 with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1b2a9be1-94a3-418b-ae63-1b24394c026b_calibrate_994767_23.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/1b2a9be1-94a3-418b-ae63-1b24394c026b_calibrate_994767_23.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34fb21cf3952f0b629b577bc85735679 with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/40ca6169-d806-4f0a-8113-5f98f183f8e6_calibrate_994767_36.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/40ca6169-d806-4f0a-8113-5f98f183f8e6_calibrate_994767_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:21 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5b993016436fb839e34d11ff800ab3a with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/e749c11d-c4ad-45be-8172-e694922e4648_calibrate_994766_53.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/e749c11d-c4ad-45be-8172-e694922e4648_calibrate_994766_53.stderr parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9c02ea094f5f5a08c2881a672c9b74f with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/bcfc2265-a6e8-4925-ba3b-e1a6382b5feb_calibrate_994767_46.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/bcfc2265-a6e8-4925-ba3b-e1a6382b5feb_calibrate_994767_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:21 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:21 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:21 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef45d256c395b9d9a21dbc949ea11b12 with result from task 661 parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:22 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash 4459bf39bf69918db5297c96c074dd2a parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/39bcbb88-ecef-4774-8d20-02faec0b6925_calibrate_994638_10.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/39bcbb88-ecef-4774-8d20-02faec0b6925_calibrate_994638_10.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/00f6fde6-c3a2-4b5e-b493-bf1e07db9015_characterizeImage_994638_10.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/00f6fde6-c3a2-4b5e-b493-bf1e07db9015_characterizeImage_994638_10.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c81ad662f45998bb48a26da75e95b5f with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 4c322287f54d78c2438109a6166a7841 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a60874b0-0ca8-4c4f-8a72-9de27f20ba75_calibrate_994765_11.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a60874b0-0ca8-4c4f-8a72-9de27f20ba75_calibrate_994765_11.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f224ba4b-334c-421e-83aa-f2dafd507a08_characterizeImage_994765_11.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/f224ba4b-334c-421e-83aa-f2dafd507a08_characterizeImage_994765_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:22 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06fe448163bbe009246ee40979568e8c with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/39830acb-b9c7-462b-bf09-1ed4765348c9_calibrate_994767_16.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/39830acb-b9c7-462b-bf09-1ed4765348c9_calibrate_994767_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:22 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acec8d2945e9d2a90784e20be962b3a6 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 60be077caf7519728686cfd1bf8382b1 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/994a0694-d6de-4c75-8825-6366fa89f8e1_calibrate_994638_4.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/994a0694-d6de-4c75-8825-6366fa89f8e1_calibrate_994638_4.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/8f6aa5d7-1845-4ae0-81bd-a16d1f435a34_characterizeImage_994638_4.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/8f6aa5d7-1845-4ae0-81bd-a16d1f435a34_characterizeImage_994638_4.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 308f3b6f161145637d172ee3b7bd9393 with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 7f1b0b94b7cec06b92e09f0d5562bf52 parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3616f757-4171-48f5-aaa1-86f51d2caf7f_calibrate_994766_51.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3616f757-4171-48f5-aaa1-86f51d2caf7f_calibrate_994766_51.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e4e9a73a-3536-47ca-9518-429430b5b8de_characterizeImage_994766_51.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e4e9a73a-3536-47ca-9518-429430b5b8de_characterizeImage_994766_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:22 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e49f509d15642667f6b6725608509bc with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 5bffa87be8c68de225db6cb816dba66e parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/278ed6a6-b2f0-44cd-98e7-fce14211f1cd_calibrate_994767_19.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/278ed6a6-b2f0-44cd-98e7-fce14211f1cd_calibrate_994767_19.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/7d4d4010-c886-4303-9e93-dbcaaf876174_characterizeImage_994767_19.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/7d4d4010-c886-4303-9e93-dbcaaf876174_characterizeImage_994767_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:22 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 607ec638c3a9ecab435ac49ad54657c4 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 40e1d76fa9b363bdba1b912e0491fa8b parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/dfda1b6f-41fe-46d2-8215-64c04dabed74_calibrate_994639_14.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/dfda1b6f-41fe-46d2-8215-64c04dabed74_calibrate_994639_14.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c55f3d39-91aa-4ee7-800e-e0e9200c3b82_characterizeImage_994639_14.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c55f3d39-91aa-4ee7-800e-e0e9200c3b82_characterizeImage_994639_14.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2944044a5dff47535182befde4701538 with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash 0fa56faa8788603289d9dcaafd0663d3 parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/10772efd-ac9b-40bd-bd40-06691d72edba_calibrate_994767_41.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/10772efd-ac9b-40bd-bd40-06691d72edba_calibrate_994767_41.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c7c5ceb5-1dd8-46c7-8e9c-ff035b868efd_characterizeImage_994767_41.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c7c5ceb5-1dd8-46c7-8e9c-ff035b868efd_characterizeImage_994767_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:22 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f0deb82663594592cf371478a665341 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 9fc0645b3122c07fb00141c9f5524b78 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/295628b7-383d-4197-8a52-e9676c959c18_calibrate_994767_40.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/295628b7-383d-4197-8a52-e9676c959c18_calibrate_994767_40.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fafd5fd3-a9e0-4e67-ad82-fd08e256774a_characterizeImage_994767_40.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/fafd5fd3-a9e0-4e67-ad82-fd08e256774a_characterizeImage_994767_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:23 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:23 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ce190f5a02ed1dbd22f05b3569a7410 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 4db15db89533bb394d5b902fb47973b0 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9ff06406-01b8-47e1-92b9-e0188816e4cb_calibrate_994639_46.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9ff06406-01b8-47e1-92b9-e0188816e4cb_calibrate_994639_46.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8173a87e-aa6e-4579-baff-6168317b540a_characterizeImage_994639_46.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8173a87e-aa6e-4579-baff-6168317b540a_characterizeImage_994639_46.stderr parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fed6937667d93e73e9327d614822188 with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/02674561-56f1-40c9-b524-f173b1a28384_calibrate_994765_28.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/02674561-56f1-40c9-b524-f173b1a28384_calibrate_994765_28.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b8303bf7226be7b3db132046eefe7b0 with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/87d55d4b-d458-4a02-a71c-4a52e94c4247_calibrate_994765_37.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/87d55d4b-d458-4a02-a71c-4a52e94c4247_calibrate_994765_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:23 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1009 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b4c30fc656a931fa63f2462a02c559d with result from task 1009 parsl.dataflow.dflow DEBUG: Task 1009 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1010 has memoization hash 6ab0d536dde71c5d52feab9b578957a5 parsl.dataflow.memoization INFO: Task 1010 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1010 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c1270a27-7144-4441-9919-9cb131b1ccc3_calibrate_994765_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c1270a27-7144-4441-9919-9cb131b1ccc3_calibrate_994765_12.stderr parsl.dataflow.dflow INFO: Standard output for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/dd612f2f-719d-4cb3-99ca-408fdd1f4d58_characterizeImage_994765_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1009 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/dd612f2f-719d-4cb3-99ca-408fdd1f4d58_characterizeImage_994765_12.stderr parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58191daddb985d30a1cd693aa0077dda with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 0f14ab1b4834e4211e09c3481a58ad2d parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65076a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/2ab6cd76-adc7-449a-81ab-1657f001408c_calibrate_994638_24.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/2ab6cd76-adc7-449a-81ab-1657f001408c_calibrate_994638_24.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/c2228bed-f33e-450a-ad05-93d422e19e07_characterizeImage_994638_24.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/c2228bed-f33e-450a-ad05-93d422e19e07_characterizeImage_994638_24.stderr parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3be0795ec61b3fe6265df1009fc92b99 with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 237dfea67f2f3b5bf302cb84d7d8daf4 parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/0f2a6f5e-753c-4a0c-92d3-10a080db6d2b_calibrate_994640_25.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/0f2a6f5e-753c-4a0c-92d3-10a080db6d2b_calibrate_994640_25.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/da68c327-1ab5-4114-9bf7-05234681ee4e_characterizeImage_994640_25.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/da68c327-1ab5-4114-9bf7-05234681ee4e_characterizeImage_994640_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:24 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 749 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3b2772fef37ae38cccfe99ef9ee2b4a with result from task 749 parsl.dataflow.dflow DEBUG: Task 749 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d24220af-6d9f-4c72-b216-d1f73d4017d3_calibrate_994767_42.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d24220af-6d9f-4c72-b216-d1f73d4017d3_calibrate_994767_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:24 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0e0b91e2525e41c8e19e53ca1d0a50c with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 8ade45f24110a3133b9563d0944f03c2 parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/cff6d55a-bda8-44ca-b8ff-b2158dfdb4f6_calibrate_994767_52.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/cff6d55a-bda8-44ca-b8ff-b2158dfdb4f6_calibrate_994767_52.stderr parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4893b070-e191-4c12-943f-d7e36dc75dd9_characterizeImage_994767_52.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4893b070-e191-4c12-943f-d7e36dc75dd9_characterizeImage_994767_52.stderr parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 945e2905c7651112bfa92fd77115fa39 with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/cab0fdcf-9626-4332-945f-84c41489cfa7_calibrate_994765_33.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/cab0fdcf-9626-4332-945f-84c41489cfa7_calibrate_994765_33.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5757d694219fb189a2663a9d3cd0a7f with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 07d9ba5cc69a668ac02d0af2a80275d0 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b1375eac-b604-41c1-9b0e-252acbfcd1d0_calibrate_994765_14.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b1375eac-b604-41c1-9b0e-252acbfcd1d0_calibrate_994765_14.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/33d6aaa2-49b3-45a8-a904-8db2d5dfcece_characterizeImage_994765_14.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/33d6aaa2-49b3-45a8-a904-8db2d5dfcece_characterizeImage_994765_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:25 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1072 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cbf151198e90665bdc22ad47bf4c626 with result from task 1072 parsl.dataflow.dflow DEBUG: Task 1072 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1073 has memoization hash 7db3bcfa541fc19f81f4af36e897b32c parsl.dataflow.memoization INFO: Task 1073 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ef1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1073 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3ea13547-cb33-447d-8732-bfb9203a83ca_calibrate_994767_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3ea13547-cb33-447d-8732-bfb9203a83ca_calibrate_994767_26.stderr parsl.dataflow.dflow INFO: Standard output for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4c223ea1-2512-4acc-b339-95f56eb07dfa_characterizeImage_994767_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1072 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/4c223ea1-2512-4acc-b339-95f56eb07dfa_characterizeImage_994767_26.stderr parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d27e5cde400ba4ef75dd793b6dbaaf9 with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8c604ef-5e3f-4785-a7f9-d5b6732975d4_calibrate_994766_35.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8c604ef-5e3f-4785-a7f9-d5b6732975d4_calibrate_994766_35.stderr parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a704c8d07cdcb05a1131da1ff0fdb3af with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c4b34b89-1679-4e4c-9193-354f192d9624_calibrate_994767_17.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/c4b34b89-1679-4e4c-9193-354f192d9624_calibrate_994767_17.stderr parsl.dataflow.dflow INFO: Task 982 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7541247290ed82e5243238adcf37b434 with result from task 982 parsl.dataflow.dflow DEBUG: Task 982 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 983 has memoization hash ba7524c59589c9c6bcd02a92740d8e99 parsl.dataflow.memoization INFO: Task 983 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 983 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/040e71f7-5db2-42e8-a709-2aa0264e14ee_calibrate_994766_24.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/040e71f7-5db2-42e8-a709-2aa0264e14ee_calibrate_994766_24.stderr parsl.dataflow.dflow INFO: Standard output for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/b298d388-4481-4b11-86e1-1b4abb14e473_characterizeImage_994766_24.stdout parsl.dataflow.dflow INFO: Standard error for task 982 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/b298d388-4481-4b11-86e1-1b4abb14e473_characterizeImage_994766_24.stderr parsl.dataflow.dflow INFO: Task 980 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ce7dce47b723f56711bcd590fa92dcb with result from task 980 parsl.dataflow.dflow DEBUG: Task 980 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f13b5ffb-b903-4386-9a66-c901e8246aa1_calibrate_994766_8.stdout parsl.dataflow.dflow INFO: Standard error for task 980 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f13b5ffb-b903-4386-9a66-c901e8246aa1_calibrate_994766_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:25 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9758c83313e357406af68be7fc1136 with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/8d2dcc2b-05a1-405b-961f-f6c7f52a1110_calibrate_994765_54.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/8d2dcc2b-05a1-405b-961f-f6c7f52a1110_calibrate_994765_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:26 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42406d75e32774eb7122fdc89992c66a with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ae5e941d-6a2f-4813-953d-db84752bc9c4_calibrate_994767_29.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ae5e941d-6a2f-4813-953d-db84752bc9c4_calibrate_994767_29.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28a9540799c981f74563ac7189b86f0c with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash eb70973578a37d389f36369f7f7d4ddb parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6b6e9ff0-fd93-4c3d-bfdc-cf9477b54002_calibrate_994639_45.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6b6e9ff0-fd93-4c3d-bfdc-cf9477b54002_calibrate_994639_45.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/87acb1a4-78ef-457b-be18-b232481d00b8_characterizeImage_994639_45.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/87acb1a4-78ef-457b-be18-b232481d00b8_characterizeImage_994639_45.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8047247ca65bb2b3a9e7a005c93da1b6 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8b9f3b3-4772-422b-8652-f651d54704f0_calibrate_994766_31.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8b9f3b3-4772-422b-8652-f651d54704f0_calibrate_994766_31.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9ed8d3815d2fd517ccd7d929116a396 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/75748dc1-7c90-4635-82a3-57ae3e6ab17d_calibrate_994765_50.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/75748dc1-7c90-4635-82a3-57ae3e6ab17d_calibrate_994765_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:26 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1033 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96c89343f6cc11e3758d94140c612351 with result from task 1033 parsl.dataflow.dflow DEBUG: Task 1033 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1034 has memoization hash 84b8cc627b3fa8176c7432da1024bc88 parsl.dataflow.memoization INFO: Task 1034 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62eccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1034 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b7f39f0d-037b-44fb-a4e9-ce200575f826_calibrate_994767_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b7f39f0d-037b-44fb-a4e9-ce200575f826_calibrate_994767_22.stderr parsl.dataflow.dflow INFO: Standard output for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/b5591803-ca16-479b-82ed-b3cbdc87b315_characterizeImage_994767_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1033 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/b5591803-ca16-479b-82ed-b3cbdc87b315_characterizeImage_994767_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:27 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03c8b96af1f50e97421172faac315c94 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 419b6a4076cae24b828dd6e3c6cf9942 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/d1552f09-1c67-4f09-952a-ace593ce964b_calibrate_994765_16.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/d1552f09-1c67-4f09-952a-ace593ce964b_calibrate_994765_16.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/66b39563-e945-416a-93e5-1345eeccb909_characterizeImage_994765_16.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/66b39563-e945-416a-93e5-1345eeccb909_characterizeImage_994765_16.stderr parsl.dataflow.dflow INFO: Task 977 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 524b1e073f17415c0409c2999f8a3d57 with result from task 977 parsl.dataflow.dflow DEBUG: Task 977 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/90a59d82-444e-4588-80b5-b7609a14ebe8_calibrate_994767_48.stdout parsl.dataflow.dflow INFO: Standard error for task 977 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/90a59d82-444e-4588-80b5-b7609a14ebe8_calibrate_994767_48.stderr parsl.dataflow.dflow INFO: Task 944 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 613cb2def77d333dd0ae2345b80399b2 with result from task 944 parsl.dataflow.dflow DEBUG: Task 944 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/6bb17817-00f5-4490-b28e-6a0a025acb1e_calibrate_994767_3.stdout parsl.dataflow.dflow INFO: Standard error for task 944 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/6bb17817-00f5-4490-b28e-6a0a025acb1e_calibrate_994767_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 279 parsl.dataflow.strategy DEBUG: Executor multi has 279 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:15:27 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:15:27 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63cfa7038618fe8aa43adf38c98bfc22 with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/72eaed7c-b588-4f2b-b546-3ae427f65a32_calibrate_994765_55.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/72eaed7c-b588-4f2b-b546-3ae427f65a32_calibrate_994765_55.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 292137538f6097e1318c1b1c3044dcc4 with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash 7a6bb3b4e862f27200118fe1ae9440c6 parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/605576f4-9aba-49f5-a273-cf0356fc68d0_calibrate_994639_54.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/605576f4-9aba-49f5-a273-cf0356fc68d0_calibrate_994639_54.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/2ab94977-dc99-475f-8deb-ad07c8d25913_characterizeImage_994639_54.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/2ab94977-dc99-475f-8deb-ad07c8d25913_characterizeImage_994639_54.stderr parsl.dataflow.dflow INFO: Task 1081 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06aa79c7efd8624c89ddf5f6b92ddcc5 with result from task 1081 parsl.dataflow.dflow DEBUG: Task 1081 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1082 has memoization hash 0552407bd58f80e594038a7f046f3fcb parsl.dataflow.memoization INFO: Task 1082 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62efb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1082 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ce636d96-b160-4e60-9a76-c55f3704a5a8_calibrate_994766_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ce636d96-b160-4e60-9a76-c55f3704a5a8_calibrate_994766_33.stderr parsl.dataflow.dflow INFO: Standard output for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/38d40ac3-a536-41b6-aa4b-e73d79b10ce8_characterizeImage_994766_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1081 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/38d40ac3-a536-41b6-aa4b-e73d79b10ce8_characterizeImage_994766_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:28 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c80f7dbfeeddde315a6f93c22c8af13e with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash bfd83c3ee9798957ef000ddb187b03ff parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63ddbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2dad4c89-df63-452f-afee-3f3e3f3ee9e0_calibrate_994767_7.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2dad4c89-df63-452f-afee-3f3e3f3ee9e0_calibrate_994767_7.stderr parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8546076c-eee2-49c9-9942-09b0f4ddc680_characterizeImage_994767_7.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8546076c-eee2-49c9-9942-09b0f4ddc680_characterizeImage_994767_7.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ed47087be1e50b00ea2d8060c710066 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 7e82f2659ea0e4f1f74daf722bd36d63 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/862997d2-7c2b-4967-8cf2-8cbe52861412_calibrate_994640_40.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/862997d2-7c2b-4967-8cf2-8cbe52861412_calibrate_994640_40.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5d5807d7-89ea-47b9-b48e-55ae38db2fe6_characterizeImage_994640_40.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5d5807d7-89ea-47b9-b48e-55ae38db2fe6_characterizeImage_994640_40.stderr parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4582a64f3d725731119698b5c9e0ebe7 with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash 233a01401ee2f85ce140d34cee9a2d3f parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/82c76429-0df8-4b40-b5dd-717aa80c3ec3_calibrate_994766_9.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/82c76429-0df8-4b40-b5dd-717aa80c3ec3_calibrate_994766_9.stderr parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/112a3be1-4aeb-4e7e-a48a-e6ce62ee40b5_characterizeImage_994766_9.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/112a3be1-4aeb-4e7e-a48a-e6ce62ee40b5_characterizeImage_994766_9.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 804bbe7adefb978db055da4d3a273fca with result from task 737 parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:28 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/bc8261bc-a02a-49f0-8099-e22c54d359b7_calibrate_994765_26.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/bc8261bc-a02a-49f0-8099-e22c54d359b7_calibrate_994765_26.stderr parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf7915937e788cc578eb1b6a5e87ef5c with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash 8eb816bb269747b3a073f0aac2f8a14b parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/970d08ea-27db-49cf-aa3a-5690296eac32_calibrate_994638_25.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/970d08ea-27db-49cf-aa3a-5690296eac32_calibrate_994638_25.stderr parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/af7d583a-8850-44fb-a07d-c3357e0bf9b8_characterizeImage_994638_25.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/af7d583a-8850-44fb-a07d-c3357e0bf9b8_characterizeImage_994638_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:29 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:29 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8ca9be470b7f398ada7fb1a796f38fa with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/2dbff8c4-972a-4698-8d7d-7e59b4cdbe18_calibrate_994766_13.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/2dbff8c4-972a-4698-8d7d-7e59b4cdbe18_calibrate_994766_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:29 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 998 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2f2a17f8a1d76f010cf9cd0d5b6b49e with result from task 998 parsl.dataflow.dflow DEBUG: Task 998 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/369cc22c-0514-45a4-8c70-3060e705a17b_calibrate_994765_43.stdout parsl.dataflow.dflow INFO: Standard error for task 998 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/369cc22c-0514-45a4-8c70-3060e705a17b_calibrate_994765_43.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 045e7b0d20b3ce1b23cf984821827abe with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 74fc1fee530d22ad79a7fff6f0fa73ed parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f7b7f899-8746-4c05-8314-b3ec6ae54d6a_calibrate_994640_52.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f7b7f899-8746-4c05-8314-b3ec6ae54d6a_calibrate_994640_52.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ca0b2d9a-6ba8-4b7d-bd66-3daa2e829e82_characterizeImage_994640_52.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ca0b2d9a-6ba8-4b7d-bd66-3daa2e829e82_characterizeImage_994640_52.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 620283d0accd7b35cbdeded63eb0fd47 with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash dbf4e25f1a77945769fdf45daeecad47 parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/59915e8d-6b0b-4213-a694-930cebf40be3_calibrate_994639_35.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/59915e8d-6b0b-4213-a694-930cebf40be3_calibrate_994639_35.stderr parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ff61e7a2-5153-4486-a98e-a58a611daf1c_characterizeImage_994639_35.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ff61e7a2-5153-4486-a98e-a58a611daf1c_characterizeImage_994639_35.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 087da587767bc2422d9b47dfa562356a with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 7ac0df43fc44cb6eadcdb983d3ea36fb parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a60c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/172a5ab7-c299-4b01-a49d-cf80e707540f_calibrate_994639_22.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/172a5ab7-c299-4b01-a49d-cf80e707540f_calibrate_994639_22.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/4c4877dc-f21a-4357-93ae-0f8b857f40ad_characterizeImage_994639_22.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/4c4877dc-f21a-4357-93ae-0f8b857f40ad_characterizeImage_994639_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:29 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e56bde084c498cf1bb15e5bd415413b with result from task 1111 parsl.dataflow.dflow DEBUG: Task 1111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1112 has memoization hash 7d4ae7b9d94756def0dafd0057cd6d5c parsl.dataflow.memoization INFO: Task 1112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1112 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a6a1efd1-db01-4837-8409-ccedf1aded1a_calibrate_994765_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a6a1efd1-db01-4837-8409-ccedf1aded1a_calibrate_994765_32.stderr parsl.dataflow.dflow INFO: Standard output for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d38bfc30-235c-4f6d-9d78-74a033186470_characterizeImage_994765_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/d38bfc30-235c-4f6d-9d78-74a033186470_characterizeImage_994765_32.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8108ec6e9f92cc5dd3e296c854d95ade with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 341a478aa337a836884a8e22015ab5d3 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/66a9cc2b-85e0-49e0-bf76-2d1d0a157767_calibrate_994639_33.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/66a9cc2b-85e0-49e0-bf76-2d1d0a157767_calibrate_994639_33.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/9634f31b-2128-4f1e-b052-dc0d6a1aef20_characterizeImage_994639_33.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/9634f31b-2128-4f1e-b052-dc0d6a1aef20_characterizeImage_994639_33.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ffe9b1c09bc5594c85159706f32507e with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b57e2608-0e8d-4cec-bd01-d824bdf60be1_calibrate_994765_45.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b57e2608-0e8d-4cec-bd01-d824bdf60be1_calibrate_994765_45.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe57d454688971f45671c0b3e87e7d1 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash d5c82cb37cb38fb407ea7b029417c88d parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/1ffbcfa9-4e76-4860-93f9-6fd81c4b7b48_calibrate_994638_30.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/1ffbcfa9-4e76-4860-93f9-6fd81c4b7b48_calibrate_994638_30.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/fa9b9872-30c6-4d2f-aa00-4a3d63a87f6d_characterizeImage_994638_30.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/fa9b9872-30c6-4d2f-aa00-4a3d63a87f6d_characterizeImage_994638_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:30 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 259317e8782f4adde6db7f16ff078e8f with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 53e8d8d929c00696935625268024f804 parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/97408f87-2201-483d-9e6d-700b5a17e2e1_calibrate_994766_3.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/97408f87-2201-483d-9e6d-700b5a17e2e1_calibrate_994766_3.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/756de5d3-d58a-4c98-8af4-2bca448ce13e_characterizeImage_994766_3.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/756de5d3-d58a-4c98-8af4-2bca448ce13e_characterizeImage_994766_3.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ced15700c178eacea6a9a60745a0cc2 with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 20840f45050b0cf88997405cdda99940 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64be700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/5aead52c-7535-4e3c-9eac-066b923c3716_calibrate_994767_35.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/5aead52c-7535-4e3c-9eac-066b923c3716_calibrate_994767_35.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/00cf9354-9dd6-429d-9713-e0ed6b2a8702_characterizeImage_994767_35.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/00cf9354-9dd6-429d-9713-e0ed6b2a8702_characterizeImage_994767_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:30 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9c7e4a32ce238d113fb05a8fe0c5094 with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b9e67ef3-c802-4342-94e6-1ea30e739630_calibrate_994766_6.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b9e67ef3-c802-4342-94e6-1ea30e739630_calibrate_994766_6.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa1d82de9084d4a1276d08d8a91b86a4 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/a5e9d841-a183-4e7d-9433-2a98a24fe62e_calibrate_994767_47.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/a5e9d841-a183-4e7d-9433-2a98a24fe62e_calibrate_994767_47.stderr parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99090aea498c3cb8cc6595e4152eb4c7 with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 00487136ce2dc476804bba42687f3814 parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/1977502e-6630-448e-9ec7-5ba14e3cb3f9_calibrate_994765_35.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/1977502e-6630-448e-9ec7-5ba14e3cb3f9_calibrate_994765_35.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/dda9ace1-c631-4efd-a366-220b46db6d2f_characterizeImage_994765_35.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/dda9ace1-c631-4efd-a366-220b46db6d2f_characterizeImage_994765_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:31 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3520e5cdfd008c702bbc34fef374bbe9 with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash 276b0d807c8ea364dd5248bddb05a3b9 parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63ded40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/c5511e0c-d81b-4657-8428-e563b9dfd308_calibrate_994766_41.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/c5511e0c-d81b-4657-8428-e563b9dfd308_calibrate_994766_41.stderr parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/cc14be5f-5733-46cf-9c61-ad8b1c199c01_characterizeImage_994766_41.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/cc14be5f-5733-46cf-9c61-ad8b1c199c01_characterizeImage_994766_41.stderr parsl.dataflow.dflow INFO: Task 950 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7b56eefee58175525f65d89faf94387 with result from task 950 parsl.dataflow.dflow DEBUG: Task 950 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/a3400530-229e-46ac-87c8-4bd8db12d5a5_calibrate_994767_59.stdout parsl.dataflow.dflow INFO: Standard error for task 950 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/a3400530-229e-46ac-87c8-4bd8db12d5a5_calibrate_994767_59.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 147d010124657f216f64447035f37a45 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f83f0258-a45c-4717-935e-69b083f187a6_calibrate_994766_50.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f83f0258-a45c-4717-935e-69b083f187a6_calibrate_994766_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:31 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c9621fa31f7fcb956f7a48d889f36c5 with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/abe18b3b-ef0f-468b-bdc8-bbdb30b65c8e_calibrate_994765_59.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/abe18b3b-ef0f-468b-bdc8-bbdb30b65c8e_calibrate_994765_59.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 066a65b75e7c080f73d8d2b7dfbe274d with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 08a480bbbb3f862abd693cde34b634a3 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b692bd61-9db3-47fd-a1cf-0b3c33f26d74_calibrate_994766_55.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b692bd61-9db3-47fd-a1cf-0b3c33f26d74_calibrate_994766_55.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a833502c-cfae-452f-b312-328db72b955a_characterizeImage_994766_55.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a833502c-cfae-452f-b312-328db72b955a_characterizeImage_994766_55.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4834ed578a62399ec4db3b8b2ac693ec with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/38210995-54ff-4d3b-bce2-425fa71f9c10_calibrate_994766_11.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/38210995-54ff-4d3b-bce2-425fa71f9c10_calibrate_994766_11.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 249eff07d0e7468b9964b0fd979a59ba with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 04a5e8881334f80b1c3dfb8745754d27 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/5544ca30-20c8-44a6-adff-76904921eac1_calibrate_994766_5.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/5544ca30-20c8-44a6-adff-76904921eac1_calibrate_994766_5.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/58a0c1ba-43b8-4444-88b5-26b426a971b7_characterizeImage_994766_5.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/58a0c1ba-43b8-4444-88b5-26b426a971b7_characterizeImage_994766_5.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efe234cab7b1b1bbbf6b8267b36f3bf8 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 93d85e4309e2f802b2fbd863005660ff parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b87a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/1f6f3c0c-c397-49a1-84c3-b2611c8d3926_calibrate_994639_12.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/1f6f3c0c-c397-49a1-84c3-b2611c8d3926_calibrate_994639_12.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/27300754-fc7e-4fb7-bf51-3cebcc49aefc_characterizeImage_994639_12.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/27300754-fc7e-4fb7-bf51-3cebcc49aefc_characterizeImage_994639_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:32 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 734 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56a1fd27e5c996a4f2426f0c2931a109 with result from task 734 parsl.dataflow.dflow DEBUG: Task 734 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/957e97d9-10da-40f9-b604-be50cf4802d9_calibrate_994765_3.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/957e97d9-10da-40f9-b604-be50cf4802d9_calibrate_994765_3.stderr parsl.dataflow.dflow INFO: Task 592 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5405e6edba159e7b865817f83e8a5e4e with result from task 592 parsl.dataflow.dflow DEBUG: Task 592 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 593 has memoization hash fe9550f9fa0f03908196a6564efdef06 parsl.dataflow.memoization INFO: Task 593 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 593 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/9af88151-c705-47fa-b5d3-5250c3922e5a_calibrate_994767_1.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/9af88151-c705-47fa-b5d3-5250c3922e5a_calibrate_994767_1.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0d449e69-98ca-4522-a95f-679fc9f80c17_characterizeImage_994767_1.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/0d449e69-98ca-4522-a95f-679fc9f80c17_characterizeImage_994767_1.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76fa0c55a29c68854f34d046642d412e with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/74784a72-c643-442a-8969-e4f0d11604d2_calibrate_994767_8.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/74784a72-c643-442a-8969-e4f0d11604d2_calibrate_994767_8.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d59113fd924ae45775da4c23df1a4631 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/bc5219a4-12bd-4b46-8111-dc6f45a62f1f_calibrate_994765_5.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/bc5219a4-12bd-4b46-8111-dc6f45a62f1f_calibrate_994765_5.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb6f60bc529cbf48b8786ac9011c1140 with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash e884c812690de6008737178875bd5196 parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/213dd25c-d9a7-4a35-ba7d-35ad5a939412_calibrate_994640_31.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/213dd25c-d9a7-4a35-ba7d-35ad5a939412_calibrate_994640_31.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/417a6a62-9c0c-4068-8e76-4aca7a28e2db_characterizeImage_994640_31.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/417a6a62-9c0c-4068-8e76-4aca7a28e2db_characterizeImage_994640_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:32 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5eca015f59dbc04e95c69563529625da with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/89e15f47-08f3-49b7-b581-f2d35ac4b6bf_calibrate_994765_1.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/89e15f47-08f3-49b7-b581-f2d35ac4b6bf_calibrate_994765_1.stderr parsl.dataflow.dflow INFO: Task 992 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74166d0d818d9f66bb11692f31598e3a with result from task 992 parsl.dataflow.dflow DEBUG: Task 992 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/32e245af-ed45-4bb4-b762-94a3bcbc8fd0_calibrate_994765_47.stdout parsl.dataflow.dflow INFO: Standard error for task 992 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/32e245af-ed45-4bb4-b762-94a3bcbc8fd0_calibrate_994765_47.stderr parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfc9b03fe1f39ae1ea15b752d9c07d51 with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 2864cef28082fe706bb8f394bd681907 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 937 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3540899a-bd08-4219-a029-4e29ca8226db_calibrate_994766_54.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3540899a-bd08-4219-a029-4e29ca8226db_calibrate_994766_54.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f06b531a-211f-4be4-94fd-3f16d16bb34a_characterizeImage_994766_54.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f06b531a-211f-4be4-94fd-3f16d16bb34a_characterizeImage_994766_54.stderr parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f25c4833b446544a814853b3bd72e81e with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash 2ca1883ad08748813de1b103a44f5888 parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 938 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ab4b0c1e-8415-4c94-83dd-0699436292a5_calibrate_994765_61.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ab4b0c1e-8415-4c94-83dd-0699436292a5_calibrate_994765_61.stderr parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/cfe2560d-6f26-499c-9626-fb1c2ab00b85_characterizeImage_994765_61.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/cfe2560d-6f26-499c-9626-fb1c2ab00b85_characterizeImage_994765_61.stderr parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbe9b5e638a8cffbc9183df46046a224 with result from task 866 parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:33 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8539292e-3ff7-4b01-a5d3-7bf98cd63d3e_calibrate_994766_29.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8539292e-3ff7-4b01-a5d3-7bf98cd63d3e_calibrate_994766_29.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f906e1acffe620a756c8747a41ce59de with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 205b92fa83d593bd38d17503155e4ccc parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6603c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 939 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/359fc618-ce8e-492e-8e46-b72b46d5894f_calibrate_994767_39.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/359fc618-ce8e-492e-8e46-b72b46d5894f_calibrate_994767_39.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c31afe7c-4b76-47e1-b5cc-1ec7cf091952_characterizeImage_994767_39.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c31afe7c-4b76-47e1-b5cc-1ec7cf091952_characterizeImage_994767_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:33 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64a0c82596fef9f07384d780ed6d2f32 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 078a2964c737eb2063116ac5659ac6fb parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66523e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 940 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/dfa8f0e6-665d-4615-a2d0-96c8d45d84db_calibrate_994640_35.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/dfa8f0e6-665d-4615-a2d0-96c8d45d84db_calibrate_994640_35.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/96212b4c-6269-4f4e-a005-bb5a41f6be5b_characterizeImage_994640_35.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/96212b4c-6269-4f4e-a005-bb5a41f6be5b_characterizeImage_994640_35.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6163bdd5dfa18fac26501b1fbf896fb4 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3824aaf5-d001-45c7-a267-7240d2dd61d9_calibrate_994766_20.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3824aaf5-d001-45c7-a267-7240d2dd61d9_calibrate_994766_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:34 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd9bb27c69ef77ef5d2549148dccaf83 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 47304ad587c6e06196d859da61b864ac parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 941 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c5505869-cda9-451b-a18f-88a9b17cc9ae_calibrate_994638_59.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c5505869-cda9-451b-a18f-88a9b17cc9ae_calibrate_994638_59.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/65495b36-7879-460a-a706-a4e34a4cc722_characterizeImage_994638_59.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/65495b36-7879-460a-a706-a4e34a4cc722_characterizeImage_994638_59.stderr parsl.dataflow.dflow INFO: Task 1022 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61e0162075c0feef24e510148d3e4c07 with result from task 1022 parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:34 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow DEBUG: Task 1022 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2de7bf58-16d9-46b1-adc6-4988d4d9d743_calibrate_994765_17.stdout parsl.dataflow.dflow INFO: Standard error for task 1022 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2de7bf58-16d9-46b1-adc6-4988d4d9d743_calibrate_994765_17.stderr parsl.dataflow.dflow INFO: Task 1006 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f24b84a96494ba4af83580b35dfec597 with result from task 1006 parsl.dataflow.dflow DEBUG: Task 1006 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1007 has memoization hash 7a5153f37f71ac2d917c6b4a6657b5b0 parsl.dataflow.memoization INFO: Task 1007 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1007 try 0 launched on executor multi with executor id 942 parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/74660b62-02f6-48c9-91b2-6e06d01c81c7_calibrate_994766_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/74660b62-02f6-48c9-91b2-6e06d01c81c7_calibrate_994766_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/3ad9bd48-3dcc-4a50-a302-1e01ae7b7d11_characterizeImage_994766_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1006 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/3ad9bd48-3dcc-4a50-a302-1e01ae7b7d11_characterizeImage_994766_42.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 336e7c76f8d0af33dad1da0faf8ac3f5 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash c0401ef1ee87d9c38fd3aacb2acc999f parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 943 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/4e4f3286-35f5-4c49-ab7e-8b0a6ccb9bb2_calibrate_994766_15.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/4e4f3286-35f5-4c49-ab7e-8b0a6ccb9bb2_calibrate_994766_15.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a14105f6-206f-4666-918a-08d07d96b699_characterizeImage_994766_15.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/a14105f6-206f-4666-918a-08d07d96b699_characterizeImage_994766_15.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c02609c1df97aed783da43b236f751f with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/05bb53a7-c9bf-4318-b11b-d3e1bf4c1e0b_calibrate_994767_43.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/05bb53a7-c9bf-4318-b11b-d3e1bf4c1e0b_calibrate_994767_43.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 852d30804cd59dcfea0aea824223118f with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash e9e94b3ffd32d08785251d1f3a58dbbf parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66032e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:35 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 944 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/72fce85d-9e3f-4a99-ae0b-ac52958bc280_calibrate_994766_57.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/72fce85d-9e3f-4a99-ae0b-ac52958bc280_calibrate_994766_57.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/77ee0189-fde6-4dc1-aaca-a5f9558207fe_characterizeImage_994766_57.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/77ee0189-fde6-4dc1-aaca-a5f9558207fe_characterizeImage_994766_57.stderr parsl.dataflow.dflow INFO: Task 1043 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 315a6c86b823b74f7d3d587f812d31cb with result from task 1043 parsl.dataflow.dflow DEBUG: Task 1043 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/770a5a5b-0d70-496c-9613-35617df475de_calibrate_994765_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1043 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/770a5a5b-0d70-496c-9613-35617df475de_calibrate_994765_13.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93b4421edcc9a2b4b0c8c45ac720e26b with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash 8110aaedf31da845a8fb8c0c466ef850 parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 945 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/3a663ccb-3b02-4083-a259-f1d741e25253_calibrate_994640_7.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/3a663ccb-3b02-4083-a259-f1d741e25253_calibrate_994640_7.stderr parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/9b6e824f-4c39-4333-a77a-3b170c7b28dc_characterizeImage_994640_7.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/9b6e824f-4c39-4333-a77a-3b170c7b28dc_characterizeImage_994640_7.stderr parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a67a3fcd28c5e6e367fc1b636caf3aff with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash 51f8729aa66ca67eab6d861486874b16 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64854e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 946 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/59437c1e-4a41-472f-95c8-f2faaac0dd51_calibrate_994638_31.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/59437c1e-4a41-472f-95c8-f2faaac0dd51_calibrate_994638_31.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e365f4f4-d6d4-4d3f-ba37-84d0867d9366_characterizeImage_994638_31.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e365f4f4-d6d4-4d3f-ba37-84d0867d9366_characterizeImage_994638_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:35 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d43c875c912cd19b0a0816ebe0b1d41e with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 57aa2d15c6bb6b1d3c8123ef2c0ef2fd parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64be3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 947 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/48660348-b2e3-4bac-b9bc-d2c16314e165_calibrate_994766_48.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/48660348-b2e3-4bac-b9bc-d2c16314e165_calibrate_994766_48.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2ba2f722-bd0d-46ad-bbbe-e6cf87667d97_characterizeImage_994766_48.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/2ba2f722-bd0d-46ad-bbbe-e6cf87667d97_characterizeImage_994766_48.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67b1e8d0395901780bc9fc33afaf5a86 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 78962a802d06451e9020587903ca6ad2 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 948 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/993fb538-ceb0-4c16-8270-f3ddde815ccc_calibrate_994639_18.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/993fb538-ceb0-4c16-8270-f3ddde815ccc_calibrate_994639_18.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ac551ee4-127c-4dc0-a8bd-b6116e8f8aa5_characterizeImage_994639_18.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ac551ee4-127c-4dc0-a8bd-b6116e8f8aa5_characterizeImage_994639_18.stderr parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b55bbd5c5a663b1a7d75dc4fc38c189 with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 5cb45ddc09b02cae2f018f40d63757ce parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6504f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 949 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/bd943c9e-d96f-4d80-a038-c1e5016b2def_calibrate_994639_50.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/bd943c9e-d96f-4d80-a038-c1e5016b2def_calibrate_994639_50.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d1b3d1bc-57fe-46ac-8579-17c081735629_characterizeImage_994639_50.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d1b3d1bc-57fe-46ac-8579-17c081735629_characterizeImage_994639_50.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eac0ad43831f78b4ef2a6222e422c6c2 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash a1a62ed02338cd7410e328147329dd7f parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 950 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b08b7497-9fff-44a1-afaa-1c40b1ddfc08_calibrate_994765_53.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b08b7497-9fff-44a1-afaa-1c40b1ddfc08_calibrate_994765_53.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/316c3d75-338f-4fd2-b6ac-a0aa2d1afa67_characterizeImage_994765_53.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/316c3d75-338f-4fd2-b6ac-a0aa2d1afa67_characterizeImage_994765_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:36 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38a3d062afba5c24d056f1b707e65e60 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b6376663-80fe-4317-a2bf-92d50b2ff5eb_calibrate_994765_9.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b6376663-80fe-4317-a2bf-92d50b2ff5eb_calibrate_994765_9.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 377cdb7b86bbfbedde7f0fe3ecf2a6f3 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 873b9748fb754bb1731f68d1e2cfe0df parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 951 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8f6caa6b-d37c-4b7e-933a-2889e517326d_calibrate_994639_15.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8f6caa6b-d37c-4b7e-933a-2889e517326d_calibrate_994639_15.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/88cb4f59-b422-4109-93f2-09046347d6b1_characterizeImage_994639_15.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/88cb4f59-b422-4109-93f2-09046347d6b1_characterizeImage_994639_15.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7318551654b675182dd985d4d46f15f with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash e5acf2364db69def3fe2b7c690670d36 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 952 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2775f449-ce7c-4942-94c8-ecc74c0e2574_calibrate_994640_27.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2775f449-ce7c-4942-94c8-ecc74c0e2574_calibrate_994640_27.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e50fc3a6-1172-4b31-a9af-1135983e2bd0_characterizeImage_994640_27.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e50fc3a6-1172-4b31-a9af-1135983e2bd0_characterizeImage_994640_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:36 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:37 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bbc489f90e963598175f14b5c2669c4 with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/553b401a-2a99-4d7c-82b6-d0d7553b49cf_calibrate_994765_42.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/553b401a-2a99-4d7c-82b6-d0d7553b49cf_calibrate_994765_42.stderr parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 232b3e71d1b16859fc7ba6321f042008 with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 829d0a04b1d5478ffc3ade09dc7e4d9f parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 953 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/9637e5a8-6bba-4b78-86f5-f389dc66ab8c_calibrate_994765_15.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/9637e5a8-6bba-4b78-86f5-f389dc66ab8c_calibrate_994765_15.stderr parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/9f5031b1-fff7-4a08-ad6e-2cb92ed76d7a_characterizeImage_994765_15.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/9f5031b1-fff7-4a08-ad6e-2cb92ed76d7a_characterizeImage_994765_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:37 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:37 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bd265e0f094b8d131af546869dab9dc with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash f87f6cb3872a69b86ba55bc3440e6eb2 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 954 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ca624eae-8f92-4d81-9680-80fe2d870708_calibrate_994638_56.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ca624eae-8f92-4d81-9680-80fe2d870708_calibrate_994638_56.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b05291aa-d900-4b70-8140-a01d4e9f75ac_characterizeImage_994638_56.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b05291aa-d900-4b70-8140-a01d4e9f75ac_characterizeImage_994638_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:37 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31f19740899f46e07442fd0742cfe18b with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 7f1af9fe689b3af4dd71f54809ecb82c parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 955 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/548bde7b-34c1-4a5c-a71e-cae725ef2907_calibrate_994638_33.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/548bde7b-34c1-4a5c-a71e-cae725ef2907_calibrate_994638_33.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/282a1115-e0ce-4093-be93-fe8b48e5c268_characterizeImage_994638_33.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/282a1115-e0ce-4093-be93-fe8b48e5c268_characterizeImage_994638_33.stderr parsl.dataflow.dflow INFO: Task 1040 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c6b679c18997615d71cdcabde14622f with result from task 1040 parsl.dataflow.dflow DEBUG: Task 1040 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2b87c2f1-0cad-4d62-b4d4-ade1cfc37a21_calibrate_994765_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1040 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/2b87c2f1-0cad-4d62-b4d4-ade1cfc37a21_calibrate_994765_29.stderr parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88269b69217f24d45853b35b348d2074 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash 80fe3a84e98a14a82aac80058c82886e parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 956 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/742d8152-1800-4d9a-ac9a-3f5c35a3e0dc_calibrate_994638_42.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/742d8152-1800-4d9a-ac9a-3f5c35a3e0dc_calibrate_994638_42.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/dca9f8ac-6826-4305-90ba-12c8c4fd47a1_characterizeImage_994638_42.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/dca9f8ac-6826-4305-90ba-12c8c4fd47a1_characterizeImage_994638_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:37 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d237256b88c21dbcf2d9b67187599c76 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 67f3d8954b508b8de98343c19cccf884 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b85f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 957 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/042b77f8-7f0e-4d7e-8b53-d488a6eb7974_calibrate_994640_30.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/042b77f8-7f0e-4d7e-8b53-d488a6eb7974_calibrate_994640_30.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5bf677e0-c7ec-4f95-9cc9-920385838dcd_characterizeImage_994640_30.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5bf677e0-c7ec-4f95-9cc9-920385838dcd_characterizeImage_994640_30.stderr parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fc9dd22a508dab1f9065593c1a8305a with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash b602c7f2009894429f7db863910cf609 parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6390680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 958 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4c35ff2e-a89d-4ac6-a417-944a13d37b71_calibrate_994640_24.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4c35ff2e-a89d-4ac6-a417-944a13d37b71_calibrate_994640_24.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/80ff2920-5805-4a16-9f9d-7c5c1906d107_characterizeImage_994640_24.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/80ff2920-5805-4a16-9f9d-7c5c1906d107_characterizeImage_994640_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:38 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1028 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7383d9df057450d469c28ade377aa604 with result from task 1028 parsl.dataflow.dflow DEBUG: Task 1028 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/96d94869-046e-421d-a6e9-c3d5f2b15042_calibrate_994765_8.stdout parsl.dataflow.dflow INFO: Standard error for task 1028 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/96d94869-046e-421d-a6e9-c3d5f2b15042_calibrate_994765_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:38 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47da7e715b8293f452a590eeeca3e150 with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f2b02d47-7da2-4a95-8c98-5711f02e07e0_calibrate_994765_24.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/f2b02d47-7da2-4a95-8c98-5711f02e07e0_calibrate_994765_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:38 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07eea748d3e65ee71cc48a4c513f87bf with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash cc146f17740d1b46e9ea284adfe54b9c parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b859e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 959 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/32ec5ce3-0f23-4329-87dc-2bb974a779e0_calibrate_994638_50.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/32ec5ce3-0f23-4329-87dc-2bb974a779e0_calibrate_994638_50.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a9de49ab-5691-441c-ab9b-1fabf0c19137_characterizeImage_994638_50.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a9de49ab-5691-441c-ab9b-1fabf0c19137_characterizeImage_994638_50.stderr parsl.dataflow.dflow INFO: Task 946 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d890dec5221d09e3d11f1f6a2ea3914 with result from task 946 parsl.dataflow.dflow DEBUG: Task 946 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 947 has memoization hash 20d492a3d9c42cc3796eb5293eccbafd parsl.dataflow.memoization INFO: Task 947 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 947 try 0 launched on executor multi with executor id 960 parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/41d07348-3ee3-4fb7-955d-f48db86dfcd7_calibrate_994766_28.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/41d07348-3ee3-4fb7-955d-f48db86dfcd7_calibrate_994766_28.stderr parsl.dataflow.dflow INFO: Standard output for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/678e3e29-3388-49b0-86ef-5fbf1076be5e_characterizeImage_994766_28.stdout parsl.dataflow.dflow INFO: Standard error for task 946 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/678e3e29-3388-49b0-86ef-5fbf1076be5e_characterizeImage_994766_28.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50da3ee24b0477a2a11b64289d985a4b with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash a5a22f17065cd874b64d5a5100e76620 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 961 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/73ba6db7-cb1b-423c-85a5-223cace7c31b_calibrate_994639_41.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/73ba6db7-cb1b-423c-85a5-223cace7c31b_calibrate_994639_41.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f4fecbb2-672f-4d9b-9cbc-c6b2b3e40dd0_characterizeImage_994639_41.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f4fecbb2-672f-4d9b-9cbc-c6b2b3e40dd0_characterizeImage_994639_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:39 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 934 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 45c5916a0f7bb0c88cddb5fe908636ec with result from task 934 parsl.dataflow.dflow DEBUG: Task 934 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 935 has memoization hash 56b41d6bad18405da57ab3f901db7c1f parsl.dataflow.memoization INFO: Task 935 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 935 try 0 launched on executor multi with executor id 962 parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4c1368d7-01b0-4fa6-8ee9-d205b337f8e5_calibrate_994765_49.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4c1368d7-01b0-4fa6-8ee9-d205b337f8e5_calibrate_994765_49.stderr parsl.dataflow.dflow INFO: Standard output for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/159161a6-c36a-48f1-ab15-3390ceb4efe3_characterizeImage_994765_49.stdout parsl.dataflow.dflow INFO: Standard error for task 934 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/159161a6-c36a-48f1-ab15-3390ceb4efe3_characterizeImage_994765_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:39 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 1018 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee74d0b1fb8a825f9259e9cf229e5473 with result from task 1018 parsl.dataflow.dflow DEBUG: Task 1018 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1019 has memoization hash cddc575cb0f5a86ee23714ea65b329c1 parsl.dataflow.memoization INFO: Task 1019 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1019 try 0 launched on executor multi with executor id 963 parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7ad13248-2307-404d-a128-8b505b129729_calibrate_994639_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7ad13248-2307-404d-a128-8b505b129729_calibrate_994639_29.stderr parsl.dataflow.dflow INFO: Standard output for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f3b92211-1f61-40bc-a47c-f96136499daf_characterizeImage_994639_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1018 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f3b92211-1f61-40bc-a47c-f96136499daf_characterizeImage_994639_29.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5080fcd95e80dd4478b8b8e52d7b09c8 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash d71883686bca88fdb047f876f804ff58 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 964 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/7aa013fb-a4f0-4a3e-a39b-dd68a7b335b3_calibrate_994638_28.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/7aa013fb-a4f0-4a3e-a39b-dd68a7b335b3_calibrate_994638_28.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/6aaee5dd-ea35-490d-9c5a-ba094687221d_characterizeImage_994638_28.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/6aaee5dd-ea35-490d-9c5a-ba094687221d_characterizeImage_994638_28.stderr parsl.dataflow.dflow INFO: Task 1049 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8297ba6052151ea834602c11eb3c3071 with result from task 1049 parsl.dataflow.dflow DEBUG: Task 1049 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c01279f1-df36-4d2c-9120-4bb8b5742ac1_calibrate_994765_56.stdout parsl.dataflow.dflow INFO: Standard error for task 1049 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c01279f1-df36-4d2c-9120-4bb8b5742ac1_calibrate_994765_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:39 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 663274d2480097cdb65e729680deb7a9 with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 8e5ef22c064fb590caee4cdbfc2f09ba parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6507e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 965 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c8e69573-146f-49df-89a0-482140e01b13_calibrate_994638_16.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c8e69573-146f-49df-89a0-482140e01b13_calibrate_994638_16.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a9092632-1ce3-47ce-be45-334f449f714f_characterizeImage_994638_16.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a9092632-1ce3-47ce-be45-334f449f714f_characterizeImage_994638_16.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a0e2c6084011e89072b78a198e184ad with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/39db03f4-e8a3-41fe-bde0-5d6b7021900e_calibrate_994639_31.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/39db03f4-e8a3-41fe-bde0-5d6b7021900e_calibrate_994639_31.stderr parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79a3f9e5bf48eb2b7ee4ab5bc0bbcc75 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/fed133ab-4f1c-484d-a29f-a848098bde60_calibrate_994766_58.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/fed133ab-4f1c-484d-a29f-a848098bde60_calibrate_994766_58.stderr parsl.dataflow.dflow INFO: Task 1102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b526231cf749509378a699a6cfa48f86 with result from task 1102 parsl.dataflow.dflow DEBUG: Task 1102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1103 has memoization hash e2b0dd7db1ff047810acc1e65babd509 parsl.dataflow.memoization INFO: Task 1103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1103 try 0 launched on executor multi with executor id 966 parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e701e2ce-5b03-4ad5-bb67-560730dad6d8_calibrate_994638_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e701e2ce-5b03-4ad5-bb67-560730dad6d8_calibrate_994638_1.stderr parsl.dataflow.dflow INFO: Standard output for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/cedd3a32-3626-4d4e-9a06-260e5358258a_characterizeImage_994638_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/cedd3a32-3626-4d4e-9a06-260e5358258a_characterizeImage_994638_1.stderr parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a65da4aa230736e2343942adfab358d with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash 617b024c917c1e992d1b861cf0e5a106 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63ddf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 967 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/82be1a71-8cfc-43d7-a8e3-9d811bd3625e_calibrate_994767_38.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/82be1a71-8cfc-43d7-a8e3-9d811bd3625e_calibrate_994767_38.stderr parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2c59e152-5f98-4bc1-bdc8-49bd64408e60_characterizeImage_994767_38.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/2c59e152-5f98-4bc1-bdc8-49bd64408e60_characterizeImage_994767_38.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb46b5db4a93350e34a5dbb8f6e00314 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 039d5f2ae257f21f68ab622082ebbba4 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 968 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4905f9b4-22bb-45a5-92a0-86b5f39b3697_calibrate_994640_58.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4905f9b4-22bb-45a5-92a0-86b5f39b3697_calibrate_994640_58.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/297069e3-3409-4025-893e-94066b537c25_characterizeImage_994640_58.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/297069e3-3409-4025-893e-94066b537c25_characterizeImage_994640_58.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3bc3d5cfb6922f4059f21804daf04ae with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/a86a9f4a-a701-422c-b2c5-30368c51b8c2_calibrate_994766_43.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/a86a9f4a-a701-422c-b2c5-30368c51b8c2_calibrate_994766_43.stderr parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5366b79f484ae8772b432390ce80dd57 with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8f221a0-209f-4d5c-91cd-3134e24c74a1_calibrate_994766_27.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f8f221a0-209f-4d5c-91cd-3134e24c74a1_calibrate_994766_27.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c11612ac14945b314471c7158361c719 with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 862d41701dc0513b3cf92d45b02b35f9 parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63900e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 969 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8ab050be-96c3-45ae-bb17-4b02f9751fca_calibrate_994639_20.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8ab050be-96c3-45ae-bb17-4b02f9751fca_calibrate_994639_20.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/96d247e5-f9a8-4295-a2bb-0a23334a25a2_characterizeImage_994639_20.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/96d247e5-f9a8-4295-a2bb-0a23334a25a2_characterizeImage_994639_20.stderr parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e7e0228b5ea04e4536265d49dac1f58 with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 2b81bd6f7c29362c0b89c52ed7258740 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6507060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 970 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/24ef6f9c-fa1b-43d9-b531-5f33005eb268_calibrate_994638_14.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/24ef6f9c-fa1b-43d9-b531-5f33005eb268_calibrate_994638_14.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/337d06e0-cce3-4d82-b88f-4d1009382e22_characterizeImage_994638_14.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/337d06e0-cce3-4d82-b88f-4d1009382e22_characterizeImage_994638_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:42 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ca1d490157eccf7cd24963a7acf4b6c with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash 7ae41cbcb13479e09fdc7272584e1890 parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 971 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/947aef51-9ff3-4f0d-805b-c074d12603ea_calibrate_994767_33.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/947aef51-9ff3-4f0d-805b-c074d12603ea_calibrate_994767_33.stderr parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c1444872-1699-4752-bb0f-0526b4478caf_characterizeImage_994767_33.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/c1444872-1699-4752-bb0f-0526b4478caf_characterizeImage_994767_33.stderr parsl.dataflow.dflow INFO: Task 1045 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5906f3436b1901075f2ed4995b4e202 with result from task 1045 parsl.dataflow.dflow DEBUG: Task 1045 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1046 has memoization hash 2360e4192d1556adb97e3c75716d748b parsl.dataflow.memoization INFO: Task 1046 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ed940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1046 try 0 launched on executor multi with executor id 972 parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ea349e77-ee96-40ef-a994-b3121f2d74e5_calibrate_994765_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ea349e77-ee96-40ef-a994-b3121f2d74e5_calibrate_994765_30.stderr parsl.dataflow.dflow INFO: Standard output for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/01c8d3dd-d764-4573-b638-c9745c38969e_characterizeImage_994765_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1045 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/01c8d3dd-d764-4573-b638-c9745c38969e_characterizeImage_994765_30.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c322287f54d78c2438109a6166a7841 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a60874b0-0ca8-4c4f-8a72-9de27f20ba75_calibrate_994765_11.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a60874b0-0ca8-4c4f-8a72-9de27f20ba75_calibrate_994765_11.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5c62a90fe0834d1e80c2fcc7ca8c21 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash 8706e81b356bd811aa86e58bcad9a2a2 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6651e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 973 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/942e8574-e6e7-4c7a-a8d3-3a986a3d4cf0_calibrate_994640_34.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/942e8574-e6e7-4c7a-a8d3-3a986a3d4cf0_calibrate_994640_34.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/089de6e3-d293-4286-846f-8e5da18d086d_characterizeImage_994640_34.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/089de6e3-d293-4286-846f-8e5da18d086d_characterizeImage_994640_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:42 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 241d60595810290fb1937d2bb4aab432 with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 40a1e82209c4becb5aace0c3159a9e61 parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6506840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 974 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/7a8f5a7d-5a10-4dc4-a1f6-c369278d879e_calibrate_994640_6.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/7a8f5a7d-5a10-4dc4-a1f6-c369278d879e_calibrate_994640_6.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ef02bd2f-2ec4-451b-b142-77bd4b4499d8_characterizeImage_994640_6.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ef02bd2f-2ec4-451b-b142-77bd4b4499d8_characterizeImage_994640_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:43 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 1109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8480fd25e6cba92325da469501885ac6 with result from task 1109 parsl.dataflow.dflow DEBUG: Task 1109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3dee29e7-9d89-4709-a404-5c86befcf59d_calibrate_994767_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3dee29e7-9d89-4709-a404-5c86befcf59d_calibrate_994767_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:43 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a34d534e7c5e45757b54ff791e66d483 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 3ccf7c389926fa938ab019d9b7917107 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b84d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 975 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/371a7754-2619-485d-a094-9a536d461f99_calibrate_994639_37.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/371a7754-2619-485d-a094-9a536d461f99_calibrate_994639_37.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0688159a-5d62-4cd9-989e-fb6130a3a139_characterizeImage_994639_37.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0688159a-5d62-4cd9-989e-fb6130a3a139_characterizeImage_994639_37.stderr parsl.dataflow.dflow INFO: Task 988 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ca7af1bbcbde544fc545ceca0615d38 with result from task 988 parsl.dataflow.dflow DEBUG: Task 988 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 989 has memoization hash bec2d4524ff30020af03777226577dc3 parsl.dataflow.memoization INFO: Task 989 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 989 try 0 launched on executor multi with executor id 976 parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/dfeeae7a-07f7-4538-86c4-edcca852749d_calibrate_994638_26.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/dfeeae7a-07f7-4538-86c4-edcca852749d_calibrate_994638_26.stderr parsl.dataflow.dflow INFO: Standard output for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/10568beb-83c2-4f24-9f8d-45ecaf803da5_characterizeImage_994638_26.stdout parsl.dataflow.dflow INFO: Standard error for task 988 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/10568beb-83c2-4f24-9f8d-45ecaf803da5_characterizeImage_994638_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:44 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d45ce1eff3e96e5ad717322ad9919e3 with result from task 830 parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8242be81-bb94-45a8-a6dc-c54a027dcf68_calibrate_994766_34.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/8242be81-bb94-45a8-a6dc-c54a027dcf68_calibrate_994766_34.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 571d58c876b640f277c2efdb00a5f498 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/c39e77b1-541c-43b1-b451-f779177a7165_calibrate_994766_30.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/c39e77b1-541c-43b1-b451-f779177a7165_calibrate_994766_30.stderr parsl.dataflow.dflow INFO: Task 1010 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ab0d536dde71c5d52feab9b578957a5 with result from task 1010 parsl.dataflow.dflow DEBUG: Task 1010 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c1270a27-7144-4441-9919-9cb131b1ccc3_calibrate_994765_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1010 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/c1270a27-7144-4441-9919-9cb131b1ccc3_calibrate_994765_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:44 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:44 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3b4c6a2dc09a8a7cc35c9f6b1388ba9 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/be68004b-4cf7-43b2-a4f7-e6831a05d729_calibrate_994767_4.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/be68004b-4cf7-43b2-a4f7-e6831a05d729_calibrate_994767_4.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f84430394ccfdbc7a3d7ebba6f3942d0 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 25fd984b18d6228407c581b7c5b0b5ef parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 977 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f8bbc0c3-4451-4899-8051-c0c0831ea01d_calibrate_994640_39.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f8bbc0c3-4451-4899-8051-c0c0831ea01d_calibrate_994640_39.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/fb748477-817e-4f2e-a9f0-e2a45d6b6b96_characterizeImage_994640_39.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/fb748477-817e-4f2e-a9f0-e2a45d6b6b96_characterizeImage_994640_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:44 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:44 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c73419e5d6453394c9c50e15dc74e29e with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 9af775a15a5b82b32259fd5ff63ffd47 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 978 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/4fe6a08a-efcb-4785-b3ae-fa3d5c4e5b57_calibrate_994639_53.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/4fe6a08a-efcb-4785-b3ae-fa3d5c4e5b57_calibrate_994639_53.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/2bdaaedc-4645-4a31-a8c5-5ff53bcb7b8b_characterizeImage_994639_53.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/2bdaaedc-4645-4a31-a8c5-5ff53bcb7b8b_characterizeImage_994639_53.stderr parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e646f26a21cd807db1e7c786fd62ec9 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash e2688a7e163f9dd1d1891bdfd7aa16ba parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 979 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4188ec4b-ebf1-4d0a-a36c-b32043ed1353_calibrate_994640_32.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4188ec4b-ebf1-4d0a-a36c-b32043ed1353_calibrate_994640_32.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/f38c87de-bcf2-4fb4-a14e-622b553eea5e_characterizeImage_994640_32.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/f38c87de-bcf2-4fb4-a14e-622b553eea5e_characterizeImage_994640_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:45 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd6ed9a160d5762d8f14d1fddd982185 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/364f84d5-570f-4398-849a-44bee4065bc5_calibrate_994766_49.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/364f84d5-570f-4398-849a-44bee4065bc5_calibrate_994766_49.stderr parsl.dataflow.dflow INFO: Task 1063 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eec7a2b833b1c7872f78fc43993d7d22 with result from task 1063 parsl.dataflow.dflow DEBUG: Task 1063 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1064 has memoization hash 6fd802daa686375f09c91de5385123ef parsl.dataflow.memoization INFO: Task 1064 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1064 try 0 launched on executor multi with executor id 980 parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c85f8af1-3279-4c5b-ab76-74a61c0e045c_calibrate_994638_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c85f8af1-3279-4c5b-ab76-74a61c0e045c_calibrate_994638_35.stderr parsl.dataflow.dflow INFO: Standard output for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ea5c66d0-930b-4646-841e-93cf99dcb49a_characterizeImage_994638_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1063 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ea5c66d0-930b-4646-841e-93cf99dcb49a_characterizeImage_994638_35.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f1b0b94b7cec06b92e09f0d5562bf52 with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3616f757-4171-48f5-aaa1-86f51d2caf7f_calibrate_994766_51.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3616f757-4171-48f5-aaa1-86f51d2caf7f_calibrate_994766_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:45 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07d9ba5cc69a668ac02d0af2a80275d0 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b1375eac-b604-41c1-9b0e-252acbfcd1d0_calibrate_994765_14.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b1375eac-b604-41c1-9b0e-252acbfcd1d0_calibrate_994765_14.stderr parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59c1aa7f7cd53b4b18f64e2c5f012d5d with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ddb29960-86e9-44ac-bf5c-178331d06d7c_calibrate_994767_32.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/ddb29960-86e9-44ac-bf5c-178331d06d7c_calibrate_994767_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:45 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29460cb733d572c5e37662f3e6e93c93 with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/fd616599-937b-479a-9854-d8a9b6bddf21_calibrate_994765_58.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/fd616599-937b-479a-9854-d8a9b6bddf21_calibrate_994765_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:46 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1601d8a69ab62bfb749f248edebfb298 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash d27b9007f1152eb7dc633d41ba0709df parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 981 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/62e289a2-fa3c-42ee-b6ed-3f4c58079e95_calibrate_994639_21.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/62e289a2-fa3c-42ee-b6ed-3f4c58079e95_calibrate_994639_21.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/74c76584-8472-444e-a19c-ad80322e505e_characterizeImage_994639_21.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/74c76584-8472-444e-a19c-ad80322e505e_characterizeImage_994639_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:47 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bffa87be8c68de225db6cb816dba66e with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/278ed6a6-b2f0-44cd-98e7-fce14211f1cd_calibrate_994767_19.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/278ed6a6-b2f0-44cd-98e7-fce14211f1cd_calibrate_994767_19.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae1ad0fe244425d7b4fefcf8c1d7b712 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 2a61e8ca349796cdeaea2dbb4b2ff7cb parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 982 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f10e2cac-e075-42cb-8dba-5b9d350a6e84_calibrate_994766_38.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f10e2cac-e075-42cb-8dba-5b9d350a6e84_calibrate_994766_38.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/3af779ef-2a16-4814-ae73-42a33c0bea62_characterizeImage_994766_38.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/3af779ef-2a16-4814-ae73-42a33c0bea62_characterizeImage_994766_38.stderr parsl.dataflow.dflow INFO: Task 1099 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 514a14d654c4984e5c4dac0f4efc303a with result from task 1099 parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:47 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow DEBUG: Task 1099 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1100 has memoization hash 08a5765a5a9eacb3379dcf060cea28bc parsl.dataflow.memoization INFO: Task 1100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63449a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1100 try 0 launched on executor multi with executor id 983 parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/eb9d520d-15ee-4d38-9635-b7e655e2c6cd_calibrate_994640_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/eb9d520d-15ee-4d38-9635-b7e655e2c6cd_calibrate_994640_36.stderr parsl.dataflow.dflow INFO: Standard output for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/78fb25cf-d016-43dc-9c40-87aaba915688_characterizeImage_994640_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1099 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/78fb25cf-d016-43dc-9c40-87aaba915688_characterizeImage_994640_36.stderr parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 048a5f4fe9938015685c5b0bc88b759d with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2213984e-d5fb-44cc-b8b4-0eacbfa73acd_calibrate_994767_58.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2213984e-d5fb-44cc-b8b4-0eacbfa73acd_calibrate_994767_58.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1de3f59e1196a0683aa802d67b774522 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash c25c53dc9dca735c3509c2545039dc5b parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 984 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/4e47a909-dee6-4e1c-9c70-61a989d3567e_calibrate_994638_48.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/4e47a909-dee6-4e1c-9c70-61a989d3567e_calibrate_994638_48.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a3e0f48f-8a9a-43c3-804d-ffdf02488b81_characterizeImage_994638_48.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a3e0f48f-8a9a-43c3-804d-ffdf02488b81_characterizeImage_994638_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:48 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cea7f8b8773e06a7ed476ce455538645 with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8bd78f96-89bc-4997-98cc-be7b3c269b26_calibrate_994767_50.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/8bd78f96-89bc-4997-98cc-be7b3c269b26_calibrate_994767_50.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 046cdc6186c5b9bc3dd3e98c67b3c5a8 with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 5e9d4ec37a824432cdaf77b526cd0a0e parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 985 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/6b9cdd78-a091-4b70-94ff-75f0d01b9ffd_calibrate_994638_21.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/6b9cdd78-a091-4b70-94ff-75f0d01b9ffd_calibrate_994638_21.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ecbef09d-7d5e-4cec-8b89-6a566a8c9d28_characterizeImage_994638_21.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ecbef09d-7d5e-4cec-8b89-6a566a8c9d28_characterizeImage_994638_21.stderr parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfacafd185aa926f78c1e5ba47042243 with result from task 745 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 0a840611731334cf0a2ee1903f623115 parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6391c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 986 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6c65ff0d-301c-4b2e-b057-fb20c39626eb_calibrate_994639_1.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6c65ff0d-301c-4b2e-b057-fb20c39626eb_calibrate_994639_1.stderr parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3b8872a1-af2c-42c2-a790-854423945540_characterizeImage_994639_1.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3b8872a1-af2c-42c2-a790-854423945540_characterizeImage_994639_1.stderr parsl.dataflow.dflow INFO: Task 1031 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6ac8ec1416a4742dac16ecb8ce3df8c with result from task 1031 parsl.dataflow.dflow DEBUG: Task 1031 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/db4104bc-36fe-4224-8ee3-7c410e772ac6_calibrate_994766_46.stdout parsl.dataflow.dflow INFO: Standard error for task 1031 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/db4104bc-36fe-4224-8ee3-7c410e772ac6_calibrate_994766_46.stderr parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02f44a2182f2f70e68bdfa27d2dbd0ac with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash ed1149ec2bdb1e9b6fb72cc028461aad parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 987 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c0554aaf-7434-4e1f-b21a-afa051f9ee33_calibrate_994639_58.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c0554aaf-7434-4e1f-b21a-afa051f9ee33_calibrate_994639_58.stderr parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0f802a9c-7951-478c-951a-f41789e31a66_characterizeImage_994639_58.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0f802a9c-7951-478c-951a-f41789e31a66_characterizeImage_994639_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:48 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:48 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9b34c941fe5df2d89ef0c05241a9fd0 with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 9b3587d665e41cefb02a315aecc910fb parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 988 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/75c7eac2-ce6e-4a3c-9d04-b6097fdc2e01_calibrate_994766_44.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/75c7eac2-ce6e-4a3c-9d04-b6097fdc2e01_calibrate_994766_44.stderr parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e99e5f49-de59-40ca-a027-fa9843974684_characterizeImage_994766_44.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/e99e5f49-de59-40ca-a027-fa9843974684_characterizeImage_994766_44.stderr parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdca1e100d0105ddd6bbc8f070b2c668 with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash 7f5821c1af79dae12be9e49a6cb6f2d8 parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 989 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d659e998-1007-4a70-9c78-b204de522f17_calibrate_994638_9.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d659e998-1007-4a70-9c78-b204de522f17_calibrate_994638_9.stderr parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9b728f80-0619-46c6-a6b1-555cea43eeeb_characterizeImage_994638_9.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9b728f80-0619-46c6-a6b1-555cea43eeeb_characterizeImage_994638_9.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 184c68f8aad3ba9cd7a6cde831d50ddb with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 54965e975a8ef08a3512890386ee92bf parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 990 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2fa2fab1-3ff1-4532-92bb-a9ae7ee84a45_calibrate_994640_41.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2fa2fab1-3ff1-4532-92bb-a9ae7ee84a45_calibrate_994640_41.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/242fbb5c-2f75-49af-8341-5707745a7dde_characterizeImage_994640_41.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/242fbb5c-2f75-49af-8341-5707745a7dde_characterizeImage_994640_41.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a6a9c27a755349e8c374466421eea1f with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash 4f80ff5e61a37299517616f408bec994 parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63de5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 991 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/82f83db5-e03c-46d5-92d4-d6e14c7574ea_calibrate_994639_10.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/82f83db5-e03c-46d5-92d4-d6e14c7574ea_calibrate_994639_10.stderr parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d510ddeb-3dae-4e43-8df3-be2383f7c5ff_characterizeImage_994639_10.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d510ddeb-3dae-4e43-8df3-be2383f7c5ff_characterizeImage_994639_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:49 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 1070 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 79062b00a9b63d3c16cf6cbf28be3505 with result from task 1070 parsl.dataflow.dflow DEBUG: Task 1070 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7202fa5a-d952-4af7-a436-86a519c91202_calibrate_994767_44.stdout parsl.dataflow.dflow INFO: Standard error for task 1070 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7202fa5a-d952-4af7-a436-86a519c91202_calibrate_994767_44.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e60d0d4b3afe3ea7d633f020c8cf4f60 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash f7aa52ab5d7c3a7438651bda641e8e94 parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 992 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/71953c32-0db4-47f8-ba43-397b2a724d0e_calibrate_994639_5.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/71953c32-0db4-47f8-ba43-397b2a724d0e_calibrate_994639_5.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/323bcf3e-aa70-4e09-874b-a817e16f6c22_characterizeImage_994639_5.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/323bcf3e-aa70-4e09-874b-a817e16f6c22_characterizeImage_994639_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:49 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 442eec3f5d280a648d2d7e3d2183c9cd with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 51d26baa3691bd8554a4f2daf5ea4dd5 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 993 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e51fae17-5c80-41b9-9b15-7608c24a7091_calibrate_994638_27.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e51fae17-5c80-41b9-9b15-7608c24a7091_calibrate_994638_27.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a77a7c89-7273-4ea6-993d-f5f1f9192ca9_characterizeImage_994638_27.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a77a7c89-7273-4ea6-993d-f5f1f9192ca9_characterizeImage_994638_27.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0172e54e45dc7df3cd3809834a67eb2 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash d1eb18211defb2d203803cd178a18dc7 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 994 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/d20a2dfe-f096-47ca-b87d-099b56ea7892_calibrate_994639_51.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/d20a2dfe-f096-47ca-b87d-099b56ea7892_calibrate_994639_51.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/277a4ab8-af3e-47c1-80c6-6c3a0c5272c9_characterizeImage_994639_51.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/277a4ab8-af3e-47c1-80c6-6c3a0c5272c9_characterizeImage_994639_51.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0bfe20ddb8a7c4d2ea5dfb3707869aa with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash cd27b43cf8db86f08024b457a089e067 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6507ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 995 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b36de761-fd3c-4478-a474-89f49fd6dc36_calibrate_994640_59.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b36de761-fd3c-4478-a474-89f49fd6dc36_calibrate_994640_59.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/558ebeb7-e54a-47f4-9e8f-1ede5c900846_characterizeImage_994640_59.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/558ebeb7-e54a-47f4-9e8f-1ede5c900846_characterizeImage_994640_59.stderr parsl.dataflow.dflow INFO: Task 1112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d4ae7b9d94756def0dafd0057cd6d5c with result from task 1112 parsl.dataflow.dflow DEBUG: Task 1112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a6a1efd1-db01-4837-8409-ccedf1aded1a_calibrate_994765_32.stdout parsl.dataflow.dflow INFO: Standard error for task 1112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/a6a1efd1-db01-4837-8409-ccedf1aded1a_calibrate_994765_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:50 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 1073 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7db3bcfa541fc19f81f4af36e897b32c with result from task 1073 parsl.dataflow.dflow DEBUG: Task 1073 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3ea13547-cb33-447d-8732-bfb9203a83ca_calibrate_994767_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1073 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3ea13547-cb33-447d-8732-bfb9203a83ca_calibrate_994767_26.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 537457696c8ec3bd42c7507b6b45916c with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/eb2ddf91-6913-4db4-a2a6-de72f71337a3_calibrate_994639_16.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/eb2ddf91-6913-4db4-a2a6-de72f71337a3_calibrate_994639_16.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47a72409193e1efa5d8b34de831afff3 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/9866727d-abe2-4db3-a961-2bbcfcf2e9dd_calibrate_994640_4.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/9866727d-abe2-4db3-a961-2bbcfcf2e9dd_calibrate_994640_4.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fc6e25749a5e3476fb5c49baafe4f3a with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash aeedced385c44dc5eed8144e54e94a4e parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 996 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/75ce4930-8a17-4fa0-a137-aa5b92d940db_calibrate_994640_21.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/75ce4930-8a17-4fa0-a137-aa5b92d940db_calibrate_994640_21.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5428fed8-d453-4108-9d88-a7a3fd1f820f_characterizeImage_994640_21.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5428fed8-d453-4108-9d88-a7a3fd1f820f_characterizeImage_994640_21.stderr parsl.dataflow.dflow INFO: Task 970 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b219695a4c622bf2085d9278cb28c72e with result from task 970 parsl.dataflow.dflow DEBUG: Task 970 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 971 has memoization hash 47e5cc4e7e8d6a619fc585e19a964db0 parsl.dataflow.memoization INFO: Task 971 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a49a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 971 try 0 launched on executor multi with executor id 997 parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ee43ae39-14b0-46c7-a53e-b6fe1e2e5950_calibrate_994638_46.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ee43ae39-14b0-46c7-a53e-b6fe1e2e5950_calibrate_994638_46.stderr parsl.dataflow.dflow INFO: Standard output for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e5c29989-6b69-4e29-a506-4254454653f9_characterizeImage_994638_46.stdout parsl.dataflow.dflow INFO: Standard error for task 970 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e5c29989-6b69-4e29-a506-4254454653f9_characterizeImage_994638_46.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ac97836031dcbc81c9e0da0971da3ea with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash ee44b6037f851c6af4f4e9dc518f5f89 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e5b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 998 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/0c7e830e-e5b3-40c4-aa62-77425cea9cb7_calibrate_994639_3.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/0c7e830e-e5b3-40c4-aa62-77425cea9cb7_calibrate_994639_3.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b6dddd21-96f1-4ed0-ac77-0024241991df_characterizeImage_994639_3.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b6dddd21-96f1-4ed0-ac77-0024241991df_characterizeImage_994639_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:50 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00487136ce2dc476804bba42687f3814 with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/1977502e-6630-448e-9ec7-5ba14e3cb3f9_calibrate_994765_35.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/1977502e-6630-448e-9ec7-5ba14e3cb3f9_calibrate_994765_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:51 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 1082 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0552407bd58f80e594038a7f046f3fcb with result from task 1082 parsl.dataflow.dflow DEBUG: Task 1082 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ce636d96-b160-4e60-9a76-c55f3704a5a8_calibrate_994766_33.stdout parsl.dataflow.dflow INFO: Standard error for task 1082 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ce636d96-b160-4e60-9a76-c55f3704a5a8_calibrate_994766_33.stderr parsl.dataflow.dflow INFO: Task 983 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba7524c59589c9c6bcd02a92740d8e99 with result from task 983 parsl.dataflow.dflow DEBUG: Task 983 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/040e71f7-5db2-42e8-a709-2aa0264e14ee_calibrate_994766_24.stdout parsl.dataflow.dflow INFO: Standard error for task 983 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/040e71f7-5db2-42e8-a709-2aa0264e14ee_calibrate_994766_24.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5fb0b28229eb91d8e5f694eed23a47e with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 5df57f077f70b1e948a9979e456322ea parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 999 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cb0426c8-affd-4b3b-9355-97485a0fcdff_calibrate_994638_54.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cb0426c8-affd-4b3b-9355-97485a0fcdff_calibrate_994638_54.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ab8b43d2-894a-4d0b-8430-d755ad1e1ff3_characterizeImage_994638_54.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/ab8b43d2-894a-4d0b-8430-d755ad1e1ff3_characterizeImage_994638_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:51 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 233a01401ee2f85ce140d34cee9a2d3f with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/82c76429-0df8-4b40-b5dd-717aa80c3ec3_calibrate_994766_9.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/82c76429-0df8-4b40-b5dd-717aa80c3ec3_calibrate_994766_9.stderr parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfd83c3ee9798957ef000ddb187b03ff with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2dad4c89-df63-452f-afee-3f3e3f3ee9e0_calibrate_994767_7.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/2dad4c89-df63-452f-afee-3f3e3f3ee9e0_calibrate_994767_7.stderr parsl.dataflow.dflow INFO: Task 1025 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941797b4ef6e1b6fbdbdc8d8d5ef2dc4 with result from task 1025 parsl.dataflow.dflow DEBUG: Task 1025 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/455f5211-b7e1-41c9-b893-ab0b54fb328e_calibrate_994766_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1025 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/455f5211-b7e1-41c9-b893-ab0b54fb328e_calibrate_994766_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:52 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:52 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9fc0645b3122c07fb00141c9f5524b78 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/295628b7-383d-4197-8a52-e9676c959c18_calibrate_994767_40.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/295628b7-383d-4197-8a52-e9676c959c18_calibrate_994767_40.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 532bccfbe55b05cedf13535c72a03d4e with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b8762ba7-8756-40bc-9538-6c4fa950e1a9_calibrate_994640_15.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b8762ba7-8756-40bc-9538-6c4fa950e1a9_calibrate_994640_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:53 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20840f45050b0cf88997405cdda99940 with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/5aead52c-7535-4e3c-9eac-066b923c3716_calibrate_994767_35.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/5aead52c-7535-4e3c-9eac-066b923c3716_calibrate_994767_35.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba370bfbbced7800b8c431438df19971 with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash b39b7ed3849985195b044a26bb8de423 parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 1000 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d1d21cab-ce6a-432d-8bf2-7d677e1f9dcd_calibrate_994767_15.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d1d21cab-ce6a-432d-8bf2-7d677e1f9dcd_calibrate_994767_15.stderr parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/ddebb8fd-4a56-4a22-83ce-13b29785affc_characterizeImage_994767_15.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/ddebb8fd-4a56-4a22-83ce-13b29785affc_characterizeImage_994767_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:53 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9711e31ff8bf1eedb67b16d41283007 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b50c7c5e-bf41-4fc8-bc64-c2dc7221852d_calibrate_994766_37.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b50c7c5e-bf41-4fc8-bc64-c2dc7221852d_calibrate_994766_37.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81f0135719293e8d34f18ab82b9fc644 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 4b803ed2bba63295e52470c31c34ad75 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a63e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 1001 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/41ea3304-9116-41cf-bc8b-3c11c18f4ce8_calibrate_994639_11.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/41ea3304-9116-41cf-bc8b-3c11c18f4ce8_calibrate_994639_11.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7787956e-f9b7-423f-9fbc-163cdca04340_characterizeImage_994639_11.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7787956e-f9b7-423f-9fbc-163cdca04340_characterizeImage_994639_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:54 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fa56faa8788603289d9dcaafd0663d3 with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/10772efd-ac9b-40bd-bd40-06691d72edba_calibrate_994767_41.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/10772efd-ac9b-40bd-bd40-06691d72edba_calibrate_994767_41.stderr parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31e9393643ac51882d80c7bfd5a22600 with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/fa9a8386-a0ca-4cca-85af-b4800c8cdd03_calibrate_994765_36.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/fa9a8386-a0ca-4cca-85af-b4800c8cdd03_calibrate_994765_36.stderr parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6618cfcaeca61589ff73ced20921e3f with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash d27aa9291e5b40bfe622839d07807db8 parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63de2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 1002 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/02bd8220-8caf-49d7-8c22-9cf453e114da_calibrate_994638_32.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/02bd8220-8caf-49d7-8c22-9cf453e114da_calibrate_994638_32.stderr parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/78b33abb-4a6e-40f2-8dcb-1a2f51dd91b5_characterizeImage_994638_32.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/78b33abb-4a6e-40f2-8dcb-1a2f51dd91b5_characterizeImage_994638_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:54 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86792a61ea73a31ccf6d279d49155f9a with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6d504038-0779-4ef8-8070-1130239edbaf_calibrate_994766_47.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/6d504038-0779-4ef8-8070-1130239edbaf_calibrate_994766_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:55 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad2c7d03cc304b2f36f7aa546f7472ce with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash b9897fb2eb00b40ca9127bbadf8020a1 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 1003 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/1e0509ec-81d7-4c0d-b2d3-15ce8b5a970b_calibrate_994638_36.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/1e0509ec-81d7-4c0d-b2d3-15ce8b5a970b_calibrate_994638_36.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/1c67a1d6-f76d-49e4-bd57-034762ec7bc2_characterizeImage_994638_36.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/1c67a1d6-f76d-49e4-bd57-034762ec7bc2_characterizeImage_994638_36.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03721241ec80991856343a02cdfe0890 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 5df61b240fdb99c19f30ba7127048a0e parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b86ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 1004 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/51f4b98e-c867-4d29-9a6e-a117e258e5d9_calibrate_994640_3.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/51f4b98e-c867-4d29-9a6e-a117e258e5d9_calibrate_994640_3.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0e0778a3-bc42-4093-988b-f6417f432a2f_characterizeImage_994640_3.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0e0778a3-bc42-4093-988b-f6417f432a2f_characterizeImage_994640_3.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eba8b1b5e85c57e7a677ac833b69da4b with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/451ef0ea-d28b-406c-84cc-341479b835ed_calibrate_994640_46.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/451ef0ea-d28b-406c-84cc-341479b835ed_calibrate_994640_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:55 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf1990d22ac524500aafad2ae6f25623 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/87eec224-9981-44ec-aba7-e935df284ad5_calibrate_994639_39.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/87eec224-9981-44ec-aba7-e935df284ad5_calibrate_994639_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:56 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 228d32d0a1d9f0aee1b942e46b160100 with result from task 910 parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash f23a3fd2d7fad03f590ee281721b0bf2 parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 1005 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4fa03677-51ff-40d9-980b-c5a529d426a9_calibrate_994640_60.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4fa03677-51ff-40d9-980b-c5a529d426a9_calibrate_994640_60.stderr parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/29f21edc-a976-486e-8681-7df49faff034_characterizeImage_994640_60.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/29f21edc-a976-486e-8681-7df49faff034_characterizeImage_994640_60.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ade45f24110a3133b9563d0944f03c2 with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/cff6d55a-bda8-44ca-b8ff-b2158dfdb4f6_calibrate_994767_52.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/cff6d55a-bda8-44ca-b8ff-b2158dfdb4f6_calibrate_994767_52.stderr parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 850ce0b8959438d370759aa2ed1637a0 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 790b4dc0144f7a368820780b607d1233 parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 1006 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ca3350e4-2081-4b8b-aad1-cdc8ee1f7aa4_calibrate_994766_17.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ca3350e4-2081-4b8b-aad1-cdc8ee1f7aa4_calibrate_994766_17.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f0a179c7-5b2e-4fe2-a7ca-f7bc0c1f7a9d_characterizeImage_994766_17.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/f0a179c7-5b2e-4fe2-a7ca-f7bc0c1f7a9d_characterizeImage_994766_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:56 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69c6b71f07d58f4f52e4a74715282f8a with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash 92ac4dea511f064b8da938a72f9ed132 parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 1007 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/516ee58b-bd2e-436b-be97-f5a6dafaf032_calibrate_994640_42.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/516ee58b-bd2e-436b-be97-f5a6dafaf032_calibrate_994640_42.stderr parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/43bfaf91-afe4-4db0-8e8f-5ffeab3e3cc7_characterizeImage_994640_42.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/43bfaf91-afe4-4db0-8e8f-5ffeab3e3cc7_characterizeImage_994640_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:57 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 940 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9fe8f6227f73ae5f2d8c2c7cdc2a12b with result from task 940 parsl.dataflow.dflow DEBUG: Task 940 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 941 has memoization hash 7602e316443e5c5ccfc73cd96573596c parsl.dataflow.memoization INFO: Task 941 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 941 try 0 launched on executor multi with executor id 1008 parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/aaefea23-455c-4937-a854-3ef527929201_calibrate_994639_55.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/aaefea23-455c-4937-a854-3ef527929201_calibrate_994639_55.stderr parsl.dataflow.dflow INFO: Standard output for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b89aea31-b52f-484d-a521-a675c676e63b_characterizeImage_994639_55.stdout parsl.dataflow.dflow INFO: Standard error for task 940 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b89aea31-b52f-484d-a521-a675c676e63b_characterizeImage_994639_55.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b086ad4497a002f05226e8c245c9b757 with result from task 631 parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 8b3458b57121052066a7aa2d22f7a6d4 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 1009 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/db43b2c4-e553-4220-a967-276a9afbf10c_calibrate_994640_9.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/db43b2c4-e553-4220-a967-276a9afbf10c_calibrate_994640_9.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8de89558-ac38-49bb-b103-3219f4c9a3c2_characterizeImage_994640_9.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8de89558-ac38-49bb-b103-3219f4c9a3c2_characterizeImage_994640_9.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c44970e1dd0737db1e0045be01ab3645 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/302dff4d-8fae-474a-bcba-685f464bc677_calibrate_994639_24.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/302dff4d-8fae-474a-bcba-685f464bc677_calibrate_994639_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:57 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08a480bbbb3f862abd693cde34b634a3 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b692bd61-9db3-47fd-a1cf-0b3c33f26d74_calibrate_994766_55.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/b692bd61-9db3-47fd-a1cf-0b3c33f26d74_calibrate_994766_55.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd56c4e1ddc5546625e98769d39f7640 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/630770c5-d86f-4243-bcf2-43ea926835c6_calibrate_994639_6.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/630770c5-d86f-4243-bcf2-43ea926835c6_calibrate_994639_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:58 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0401ef1ee87d9c38fd3aacb2acc999f with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/4e4f3286-35f5-4c49-ab7e-8b0a6ccb9bb2_calibrate_994766_15.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/4e4f3286-35f5-4c49-ab7e-8b0a6ccb9bb2_calibrate_994766_15.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06073525f5a05a92fc4b9296bf5ca14c with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 67079c893ae4d9905ab1886286f79a69 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e4ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 1010 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/40a66c23-1c65-4361-bd3c-c06600a2bde4_calibrate_994639_59.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/40a66c23-1c65-4361-bd3c-c06600a2bde4_calibrate_994639_59.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8b6a0fac-30da-4ca7-bb00-298409a0de73_characterizeImage_994639_59.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8b6a0fac-30da-4ca7-bb00-298409a0de73_characterizeImage_994639_59.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40e1d76fa9b363bdba1b912e0491fa8b with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/dfda1b6f-41fe-46d2-8215-64c04dabed74_calibrate_994639_14.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/dfda1b6f-41fe-46d2-8215-64c04dabed74_calibrate_994639_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:59 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57aa2d15c6bb6b1d3c8123ef2c0ef2fd with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/48660348-b2e3-4bac-b9bc-d2c16314e165_calibrate_994766_48.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/48660348-b2e3-4bac-b9bc-d2c16314e165_calibrate_994766_48.stderr parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04a5e8881334f80b1c3dfb8745754d27 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/5544ca30-20c8-44a6-adff-76904921eac1_calibrate_994766_5.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/5544ca30-20c8-44a6-adff-76904921eac1_calibrate_994766_5.stderr parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2864cef28082fe706bb8f394bd681907 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3540899a-bd08-4219-a029-4e29ca8226db_calibrate_994766_54.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/3540899a-bd08-4219-a029-4e29ca8226db_calibrate_994766_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:15:59 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1060 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77975428e412b727eb8be987ccfd04a0 with result from task 1060 parsl.dataflow.dflow DEBUG: Task 1060 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1061 has memoization hash b92424a3aab49ece8b6630855b60bd80 parsl.dataflow.memoization INFO: Task 1061 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ee8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1061 try 0 launched on executor multi with executor id 1011 parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/acec8394-f794-4eb5-b80c-47082e1024a2_calibrate_994767_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/acec8394-f794-4eb5-b80c-47082e1024a2_calibrate_994767_12.stderr parsl.dataflow.dflow INFO: Standard output for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8439cb46-0a01-4611-a40a-4ecacd70d3fd_characterizeImage_994767_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1060 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/8439cb46-0a01-4611-a40a-4ecacd70d3fd_characterizeImage_994767_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:00 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1a62ed02338cd7410e328147329dd7f with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b08b7497-9fff-44a1-afaa-1c40b1ddfc08_calibrate_994765_53.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/b08b7497-9fff-44a1-afaa-1c40b1ddfc08_calibrate_994765_53.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25789bab875958374445531eb7039e1e with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash be33e4e577ce87f215d84b3d2ccee174 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 1012 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/15f4b262-4f2f-405e-a08d-a276953a924d_calibrate_994639_57.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/15f4b262-4f2f-405e-a08d-a276953a924d_calibrate_994639_57.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/4cfca11e-e4ed-4449-9e1b-4882ca7dfdee_characterizeImage_994639_57.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/4cfca11e-e4ed-4449-9e1b-4882ca7dfdee_characterizeImage_994639_57.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dde79f9526e970abad88de02da4e185d with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/90b566e7-895d-4b29-b4b9-1662ae18885b_calibrate_994638_15.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/90b566e7-895d-4b29-b4b9-1662ae18885b_calibrate_994638_15.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecf58297fca2016453f0805511f025d5 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash f1a324f5b5dfded4c0b8b635b75981ab parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a6a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 1013 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/64ca9717-a5e6-48ec-94fd-bd04196b6423_calibrate_994638_17.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/64ca9717-a5e6-48ec-94fd-bd04196b6423_calibrate_994638_17.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/53d7c85c-f3e3-4982-a721-f1581ccce5c9_characterizeImage_994638_17.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/53d7c85c-f3e3-4982-a721-f1581ccce5c9_characterizeImage_994638_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:00 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53e8d8d929c00696935625268024f804 with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/97408f87-2201-483d-9e6d-700b5a17e2e1_calibrate_994766_3.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/97408f87-2201-483d-9e6d-700b5a17e2e1_calibrate_994766_3.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 341a478aa337a836884a8e22015ab5d3 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/66a9cc2b-85e0-49e0-bf76-2d1d0a157767_calibrate_994639_33.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/66a9cc2b-85e0-49e0-bf76-2d1d0a157767_calibrate_994639_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:01 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e884c812690de6008737178875bd5196 with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/213dd25c-d9a7-4a35-ba7d-35ad5a939412_calibrate_994640_31.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/213dd25c-d9a7-4a35-ba7d-35ad5a939412_calibrate_994640_31.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9e94b3ffd32d08785251d1f3a58dbbf with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/72fce85d-9e3f-4a99-ae0b-ac52958bc280_calibrate_994766_57.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/72fce85d-9e3f-4a99-ae0b-ac52958bc280_calibrate_994766_57.stderr parsl.dataflow.dflow INFO: Task 935 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56b41d6bad18405da57ab3f901db7c1f with result from task 935 parsl.dataflow.dflow DEBUG: Task 935 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4c1368d7-01b0-4fa6-8ee9-d205b337f8e5_calibrate_994765_49.stdout parsl.dataflow.dflow INFO: Standard error for task 935 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/4c1368d7-01b0-4fa6-8ee9-d205b337f8e5_calibrate_994765_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:01 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 931 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cafbd8d59aff5c3878c8d1b18d50a64 with result from task 931 parsl.dataflow.dflow DEBUG: Task 931 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 932 has memoization hash 74b9b4af72d4da99dc5116e4c746cd9e parsl.dataflow.memoization INFO: Task 932 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 932 try 0 launched on executor multi with executor id 1014 parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/494e52ec-d4d5-476c-a306-f660cd6d6937_calibrate_994765_46.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/494e52ec-d4d5-476c-a306-f660cd6d6937_calibrate_994765_46.stderr parsl.dataflow.dflow INFO: Standard output for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/966d854a-e15d-46b3-b75a-297272c4309a_characterizeImage_994765_46.stdout parsl.dataflow.dflow INFO: Standard error for task 931 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994765/966d854a-e15d-46b3-b75a-297272c4309a_characterizeImage_994765_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:01 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f14ab1b4834e4211e09c3481a58ad2d with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/2ab6cd76-adc7-449a-81ab-1657f001408c_calibrate_994638_24.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/2ab6cd76-adc7-449a-81ab-1657f001408c_calibrate_994638_24.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bb4e1a48d1125f9debe1d51bb63bf37 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b1b7f472-e155-4de9-9cde-ba507bf3792c_calibrate_994767_10.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b1b7f472-e155-4de9-9cde-ba507bf3792c_calibrate_994767_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:02 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d994abe82a78477079178fd46f3ed7f with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 743aab7ff0322b5fccf7a14baa4fdfd8 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a5da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 1015 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/eab36fd5-92d3-4b02-9b77-14878a58702d_calibrate_994640_62.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/eab36fd5-92d3-4b02-9b77-14878a58702d_calibrate_994640_62.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/cb63d270-880f-4f7f-9d3d-c25edae6250b_characterizeImage_994640_62.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/cb63d270-880f-4f7f-9d3d-c25edae6250b_characterizeImage_994640_62.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 205b92fa83d593bd38d17503155e4ccc with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/359fc618-ce8e-492e-8e46-b72b46d5894f_calibrate_994767_39.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/359fc618-ce8e-492e-8e46-b72b46d5894f_calibrate_994767_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:02 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f50db04d162e117d5e8dd49a7489313 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash c31d6f6c8664f073ed19ab863ff13b0d parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 1016 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4b7a741d-e5c7-4ffc-b5fa-bfec48c6880b_calibrate_994640_37.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4b7a741d-e5c7-4ffc-b5fa-bfec48c6880b_calibrate_994640_37.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8ef89b07-43ff-48f4-993b-ad655ad25f11_characterizeImage_994640_37.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8ef89b07-43ff-48f4-993b-ad655ad25f11_characterizeImage_994640_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:03 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c52e5172887e423b8df29b69212055e with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 471464f17e6b86082cd5a01940afae9e parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6486de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 1017 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/0b4c133b-78c9-483d-8d7e-6e3e038d8eb8_calibrate_994638_6.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/0b4c133b-78c9-483d-8d7e-6e3e038d8eb8_calibrate_994638_6.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/c6e42758-3864-4a66-9cf6-2be6955d4ebb_characterizeImage_994638_6.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/c6e42758-3864-4a66-9cf6-2be6955d4ebb_characterizeImage_994638_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:03 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb70973578a37d389f36369f7f7d4ddb with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6b6e9ff0-fd93-4c3d-bfdc-cf9477b54002_calibrate_994639_45.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6b6e9ff0-fd93-4c3d-bfdc-cf9477b54002_calibrate_994639_45.stderr parsl.dataflow.dflow INFO: Task 994 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 203f6b4f0748590d700e70c02bc88937 with result from task 994 parsl.dataflow.dflow DEBUG: Task 994 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 995 has memoization hash 6ff00a58790a2f046a1ccc5a6dc88142 parsl.dataflow.memoization INFO: Task 995 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ec220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 995 try 0 launched on executor multi with executor id 1018 parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/f5b1855e-1e27-472b-a912-8be71cad9fd7_calibrate_994638_51.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/f5b1855e-1e27-472b-a912-8be71cad9fd7_calibrate_994638_51.stderr parsl.dataflow.dflow INFO: Standard output for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/fd41a028-0d17-422c-b567-a2bb9049b620_characterizeImage_994638_51.stdout parsl.dataflow.dflow INFO: Standard error for task 994 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/fd41a028-0d17-422c-b567-a2bb9049b620_characterizeImage_994638_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:03 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07ab4612d148501f926266b40047d6a4 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 89a1e997e8d55c1e3d1ad9ffcec7fe0e parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df7b84e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 1019 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c3fe9734-a55d-40f2-a492-3fc8e7521164_calibrate_994638_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c3fe9734-a55d-40f2-a492-3fc8e7521164_calibrate_994638_3.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/01f51214-cdf8-466f-81bb-8f28c60fb8eb_characterizeImage_994638_3.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/01f51214-cdf8-466f-81bb-8f28c60fb8eb_characterizeImage_994638_3.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12ee3bb1570f12ab04f34fc3efd69ae1 with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/e778cd77-a791-4268-a9dc-0a87279770cd_calibrate_994766_1.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/e778cd77-a791-4268-a9dc-0a87279770cd_calibrate_994766_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:04 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ae41cbcb13479e09fdc7272584e1890 with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/947aef51-9ff3-4f0d-805b-c074d12603ea_calibrate_994767_33.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/947aef51-9ff3-4f0d-805b-c074d12603ea_calibrate_994767_33.stderr parsl.dataflow.dflow INFO: Task 947 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20d492a3d9c42cc3796eb5293eccbafd with result from task 947 parsl.dataflow.dflow DEBUG: Task 947 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/41d07348-3ee3-4fb7-955d-f48db86dfcd7_calibrate_994766_28.stdout parsl.dataflow.dflow INFO: Standard error for task 947 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/41d07348-3ee3-4fb7-955d-f48db86dfcd7_calibrate_994766_28.stderr parsl.dataflow.dflow INFO: Task 1012 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38423bbcaeeca94a76b40e2a0f6f8a25 with result from task 1012 parsl.dataflow.dflow DEBUG: Task 1012 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1013 has memoization hash b03e2e92a2e17004ec4d5125d41de14f parsl.dataflow.memoization INFO: Task 1013 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a7100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1013 try 0 launched on executor multi with executor id 1020 parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/341363f1-50ce-4a08-afdf-bf6610284e9d_calibrate_994639_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/341363f1-50ce-4a08-afdf-bf6610284e9d_calibrate_994639_4.stderr parsl.dataflow.dflow INFO: Standard output for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d5d8e699-3acc-4f95-8001-a078b45fbe09_characterizeImage_994639_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1012 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/d5d8e699-3acc-4f95-8001-a078b45fbe09_characterizeImage_994639_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:04 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5c82cb37cb38fb407ea7b029417c88d with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/1ffbcfa9-4e76-4860-93f9-6fd81c4b7b48_calibrate_994638_30.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/1ffbcfa9-4e76-4860-93f9-6fd81c4b7b48_calibrate_994638_30.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60be077caf7519728686cfd1bf8382b1 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/994a0694-d6de-4c75-8825-6366fa89f8e1_calibrate_994638_4.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/994a0694-d6de-4c75-8825-6366fa89f8e1_calibrate_994638_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:05 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b89dfe2375a21cfdac98e9c830239746 with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash cf7ac87cde2d337cad684582d4b28cc4 parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 1021 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9e7a7155-8669-4c5b-9ceb-a87141fded12_calibrate_994639_40.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9e7a7155-8669-4c5b-9ceb-a87141fded12_calibrate_994639_40.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0fea87bc-ce23-41d4-acbc-6d11002decf6_characterizeImage_994639_40.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/0fea87bc-ce23-41d4-acbc-6d11002decf6_characterizeImage_994639_40.stderr parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 237dfea67f2f3b5bf302cb84d7d8daf4 with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/0f2a6f5e-753c-4a0c-92d3-10a080db6d2b_calibrate_994640_25.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/0f2a6f5e-753c-4a0c-92d3-10a080db6d2b_calibrate_994640_25.stderr parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60d33a40bb748dfef94b7e42c5526228 with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash 6adc6e687ffbe15fea7cf3a144adec64 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65079c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 1022 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b8d62ae9-8553-485a-b82b-438fb77a64f1_calibrate_994640_8.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b8d62ae9-8553-485a-b82b-438fb77a64f1_calibrate_994640_8.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/216637a7-c158-43dd-8f69-4c67bf086769_characterizeImage_994640_8.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/216637a7-c158-43dd-8f69-4c67bf086769_characterizeImage_994640_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:05 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a6bb3b4e862f27200118fe1ae9440c6 with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/605576f4-9aba-49f5-a273-cf0356fc68d0_calibrate_994639_54.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/605576f4-9aba-49f5-a273-cf0356fc68d0_calibrate_994639_54.stderr parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c12316fffd7e4bf3b99365c8592ad247 with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/ad519186-57a3-4cf4-aba0-fb95ae8d9281_calibrate_994640_33.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/ad519186-57a3-4cf4-aba0-fb95ae8d9281_calibrate_994640_33.stderr parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbf4e25f1a77945769fdf45daeecad47 with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/59915e8d-6b0b-4213-a694-930cebf40be3_calibrate_994639_35.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/59915e8d-6b0b-4213-a694-930cebf40be3_calibrate_994639_35.stderr parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51f8729aa66ca67eab6d861486874b16 with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/59437c1e-4a41-472f-95c8-f2faaac0dd51_calibrate_994638_31.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/59437c1e-4a41-472f-95c8-f2faaac0dd51_calibrate_994638_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:06 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61d93563a8dcb030be9b334511e68b7d with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash f9e4743cfb8eeaa7f88e5484acb0dace parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 1023 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/a879c77e-cc39-460f-ab77-2e34d27573e5_calibrate_994766_12.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/a879c77e-cc39-460f-ab77-2e34d27573e5_calibrate_994766_12.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/67e0afd5-d295-4762-8823-3dec2f32cc34_characterizeImage_994766_12.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/67e0afd5-d295-4762-8823-3dec2f32cc34_characterizeImage_994766_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:06 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:07 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:07 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ac0df43fc44cb6eadcdb983d3ea36fb with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/172a5ab7-c299-4b01-a49d-cf80e707540f_calibrate_994639_22.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/172a5ab7-c299-4b01-a49d-cf80e707540f_calibrate_994639_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:07 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:08 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1046 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2360e4192d1556adb97e3c75716d748b with result from task 1046 parsl.dataflow.dflow DEBUG: Task 1046 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ea349e77-ee96-40ef-a994-b3121f2d74e5_calibrate_994765_30.stdout parsl.dataflow.dflow INFO: Standard error for task 1046 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ea349e77-ee96-40ef-a994-b3121f2d74e5_calibrate_994765_30.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9100e25b96fede540787daf3fdfe49a9 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash bb8e75f3d8c8bd4670e39f3ec6d73383 parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 1024 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/8acb7000-e2c2-406b-85dd-3c918c5c825a_calibrate_994638_22.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/8acb7000-e2c2-406b-85dd-3c918c5c825a_calibrate_994638_22.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/cb79766a-e1d8-4355-a4f2-45b9c67c9190_characterizeImage_994638_22.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/cb79766a-e1d8-4355-a4f2-45b9c67c9190_characterizeImage_994638_22.stderr parsl.dataflow.dflow INFO: Task 1007 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a5153f37f71ac2d917c6b4a6657b5b0 with result from task 1007 parsl.dataflow.dflow DEBUG: Task 1007 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/74660b62-02f6-48c9-91b2-6e06d01c81c7_calibrate_994766_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1007 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/74660b62-02f6-48c9-91b2-6e06d01c81c7_calibrate_994766_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:08 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 180 parsl.dataflow.strategy DEBUG: Executor multi has 180 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:16:09 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:16:09 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1034 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84b8cc627b3fa8176c7432da1024bc88 with result from task 1034 parsl.dataflow.dflow DEBUG: Task 1034 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b7f39f0d-037b-44fb-a4e9-ce200575f826_calibrate_994767_22.stdout parsl.dataflow.dflow INFO: Standard error for task 1034 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/b7f39f0d-037b-44fb-a4e9-ce200575f826_calibrate_994767_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:09 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de1f3c9f75561e18cad5d3b9ec535b6f with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/06131462-475d-41db-993d-0e9b04908e72_calibrate_994765_4.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/06131462-475d-41db-993d-0e9b04908e72_calibrate_994765_4.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f1af9fe689b3af4dd71f54809ecb82c with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/548bde7b-34c1-4a5c-a71e-cae725ef2907_calibrate_994638_33.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/548bde7b-34c1-4a5c-a71e-cae725ef2907_calibrate_994638_33.stderr parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10c6fa3bc7fa469b6f5b8837716544d9 with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash c1bc025d1733de287ec789f5a7290b2a parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 1025 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/db218131-767e-40d0-afa0-22f45d7a0d93_calibrate_994767_13.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/db218131-767e-40d0-afa0-22f45d7a0d93_calibrate_994767_13.stderr parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/1a760cff-0631-4fdd-a769-a1e3245df1bb_characterizeImage_994767_13.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/1a760cff-0631-4fdd-a769-a1e3245df1bb_characterizeImage_994767_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:09 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:10 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 593 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe9550f9fa0f03908196a6564efdef06 with result from task 593 parsl.dataflow.dflow DEBUG: Task 593 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/9af88151-c705-47fa-b5d3-5250c3922e5a_calibrate_994767_1.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/9af88151-c705-47fa-b5d3-5250c3922e5a_calibrate_994767_1.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 419b6a4076cae24b828dd6e3c6cf9942 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/d1552f09-1c67-4f09-952a-ace593ce964b_calibrate_994765_16.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/d1552f09-1c67-4f09-952a-ace593ce964b_calibrate_994765_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:10 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4db15db89533bb394d5b902fb47973b0 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9ff06406-01b8-47e1-92b9-e0188816e4cb_calibrate_994639_46.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9ff06406-01b8-47e1-92b9-e0188816e4cb_calibrate_994639_46.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b301f85cabfb12fb6e670c62724edd8 with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 84adfd2e18e8f91dea51693108a9d49e parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 1026 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/fc4434dc-37ae-4951-984b-d3a5477a0e55_calibrate_994640_16.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/fc4434dc-37ae-4951-984b-d3a5477a0e55_calibrate_994640_16.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/d4f06e18-e9aa-4239-9a3d-4bcff67c53a8_characterizeImage_994640_16.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/d4f06e18-e9aa-4239-9a3d-4bcff67c53a8_characterizeImage_994640_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:10 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:10 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:10 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 617b024c917c1e992d1b861cf0e5a106 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/82be1a71-8cfc-43d7-a8e3-9d811bd3625e_calibrate_994767_38.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/82be1a71-8cfc-43d7-a8e3-9d811bd3625e_calibrate_994767_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:11 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:11 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:11 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:11 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 829d0a04b1d5478ffc3ade09dc7e4d9f with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/9637e5a8-6bba-4b78-86f5-f389dc66ab8c_calibrate_994765_15.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/9637e5a8-6bba-4b78-86f5-f389dc66ab8c_calibrate_994765_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:11 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:11 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:11 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74fc1fee530d22ad79a7fff6f0fa73ed with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f7b7f899-8746-4c05-8314-b3ec6ae54d6a_calibrate_994640_52.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f7b7f899-8746-4c05-8314-b3ec6ae54d6a_calibrate_994640_52.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67f3d8954b508b8de98343c19cccf884 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/042b77f8-7f0e-4d7e-8b53-d488a6eb7974_calibrate_994640_30.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/042b77f8-7f0e-4d7e-8b53-d488a6eb7974_calibrate_994640_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:11 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78962a802d06451e9020587903ca6ad2 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/993fb538-ceb0-4c16-8270-f3ddde815ccc_calibrate_994639_18.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/993fb538-ceb0-4c16-8270-f3ddde815ccc_calibrate_994639_18.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5acf2364db69def3fe2b7c690670d36 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2775f449-ce7c-4942-94c8-ecc74c0e2574_calibrate_994640_27.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2775f449-ce7c-4942-94c8-ecc74c0e2574_calibrate_994640_27.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 873b9748fb754bb1731f68d1e2cfe0df with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8f6caa6b-d37c-4b7e-933a-2889e517326d_calibrate_994639_15.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8f6caa6b-d37c-4b7e-933a-2889e517326d_calibrate_994639_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:13 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:13 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:14 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:14 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:14 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:14 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:14 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:14 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d71883686bca88fdb047f876f804ff58 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/7aa013fb-a4f0-4a3e-a39b-dd68a7b335b3_calibrate_994638_28.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/7aa013fb-a4f0-4a3e-a39b-dd68a7b335b3_calibrate_994638_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:14 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47304ad587c6e06196d859da61b864ac with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c5505869-cda9-451b-a18f-88a9b17cc9ae_calibrate_994638_59.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c5505869-cda9-451b-a18f-88a9b17cc9ae_calibrate_994638_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:14 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:15 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:15 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e5ef22c064fb590caee4cdbfc2f09ba with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c8e69573-146f-49df-89a0-482140e01b13_calibrate_994638_16.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c8e69573-146f-49df-89a0-482140e01b13_calibrate_994638_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:15 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:15 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:15 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5cb45ddc09b02cae2f018f40d63757ce with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/bd943c9e-d96f-4d80-a038-c1e5016b2def_calibrate_994639_50.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/bd943c9e-d96f-4d80-a038-c1e5016b2def_calibrate_994639_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:15 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e82f2659ea0e4f1f74daf722bd36d63 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/862997d2-7c2b-4967-8cf2-8cbe52861412_calibrate_994640_40.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/862997d2-7c2b-4967-8cf2-8cbe52861412_calibrate_994640_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:15 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:16 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:16 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:16 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 989 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bec2d4524ff30020af03777226577dc3 with result from task 989 parsl.dataflow.dflow DEBUG: Task 989 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/dfeeae7a-07f7-4538-86c4-edcca852749d_calibrate_994638_26.stdout parsl.dataflow.dflow INFO: Standard error for task 989 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/dfeeae7a-07f7-4538-86c4-edcca852749d_calibrate_994638_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:16 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ef077c54381c6fab3893b24a1f415b3 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 805966df981e518620ead2f90f60dd5d parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 1027 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7949719f-15f3-4ae0-98e8-5947bb4e74df_calibrate_994767_49.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7949719f-15f3-4ae0-98e8-5947bb4e74df_calibrate_994767_49.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/847de146-552e-41c0-a879-16150e744761_characterizeImage_994767_49.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/847de146-552e-41c0-a879-16150e744761_characterizeImage_994767_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:16 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 1103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2b0dd7db1ff047810acc1e65babd509 with result from task 1103 parsl.dataflow.dflow DEBUG: Task 1103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e701e2ce-5b03-4ad5-bb67-560730dad6d8_calibrate_994638_1.stdout parsl.dataflow.dflow INFO: Standard error for task 1103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e701e2ce-5b03-4ad5-bb67-560730dad6d8_calibrate_994638_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:16 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:16 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8110aaedf31da845a8fb8c0c466ef850 with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/3a663ccb-3b02-4083-a259-f1d741e25253_calibrate_994640_7.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/3a663ccb-3b02-4083-a259-f1d741e25253_calibrate_994640_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:16 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc146f17740d1b46e9ea284adfe54b9c with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/32ec5ce3-0f23-4329-87dc-2bb974a779e0_calibrate_994638_50.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/32ec5ce3-0f23-4329-87dc-2bb974a779e0_calibrate_994638_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:17 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:17 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd8cef8f385d3946255d95121e42d8e5 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash a2d4cad2b3c7879346d9ffec5837aa15 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 1028 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/102e28ee-a1ac-4614-8810-cf6bdc220049_calibrate_994638_60.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/102e28ee-a1ac-4614-8810-cf6bdc220049_calibrate_994638_60.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/bbf532fd-b578-45f7-acf6-ad4fc7bd1378_characterizeImage_994638_60.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/bbf532fd-b578-45f7-acf6-ad4fc7bd1378_characterizeImage_994638_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:17 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:17 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:17 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b39b7ed3849985195b044a26bb8de423 with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d1d21cab-ce6a-432d-8bf2-7d677e1f9dcd_calibrate_994767_15.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/d1d21cab-ce6a-432d-8bf2-7d677e1f9dcd_calibrate_994767_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:17 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:17 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:17 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e7eb8ff3f3e039542466608223b367e with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash c8bef757e2eb244b33f4c32b43e5f615 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6505da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 1029 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/02662a13-c59d-4982-af22-38581b4117c9_calibrate_994638_20.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/02662a13-c59d-4982-af22-38581b4117c9_calibrate_994638_20.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/6a1eaef0-ee48-4ee4-bc6e-a36ef201d0f3_characterizeImage_994638_20.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/6a1eaef0-ee48-4ee4-bc6e-a36ef201d0f3_characterizeImage_994638_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:18 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:18 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:18 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:18 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:18 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:18 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:18 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:19 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:19 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:19 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:19 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:19 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2688a7e163f9dd1d1891bdfd7aa16ba with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4188ec4b-ebf1-4d0a-a36c-b32043ed1353_calibrate_994640_32.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4188ec4b-ebf1-4d0a-a36c-b32043ed1353_calibrate_994640_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:19 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:19 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:19 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:20 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:20 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:20 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:20 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25fd984b18d6228407c581b7c5b0b5ef with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f8bbc0c3-4451-4899-8051-c0c0831ea01d_calibrate_994640_39.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f8bbc0c3-4451-4899-8051-c0c0831ea01d_calibrate_994640_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:20 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:20 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 513953b5b751ea11f04d65e992723e9f with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 2f1cc87624f4fe97616f01ba10ba8dcf parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6487420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 1030 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/e05cd477-65b1-4301-ae31-94f6070f3fc7_calibrate_994640_17.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/e05cd477-65b1-4301-ae31-94f6070f3fc7_calibrate_994640_17.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8f0ef8bb-3959-483b-9c98-04c78c7577e6_characterizeImage_994640_17.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/8f0ef8bb-3959-483b-9c98-04c78c7577e6_characterizeImage_994640_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:20 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:20 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:21 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:21 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9a5b53d04f58b9036973fb6d5c72c21 with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 97b0512550fa1589cfdffc2ab06de4c5 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 1031 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/e4db660d-c598-4daf-bf51-46e8f185488f_calibrate_994639_8.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/e4db660d-c598-4daf-bf51-46e8f185488f_calibrate_994639_8.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/750e181e-3b75-4420-bb25-eefe43be964a_characterizeImage_994639_8.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/750e181e-3b75-4420-bb25-eefe43be964a_characterizeImage_994639_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:21 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 209696e1c07c024d1f5df2c70ac5d30c with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash c21c7e83ba881c4fb228e290fe7c8a21 parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 1032 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/b75b0834-6c42-4fc1-88ea-93181bfc04b3_calibrate_994638_45.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/b75b0834-6c42-4fc1-88ea-93181bfc04b3_calibrate_994638_45.stderr parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e36c20c3-cbf0-4f4d-86f0-aa8088dcc063_characterizeImage_994638_45.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e36c20c3-cbf0-4f4d-86f0-aa8088dcc063_characterizeImage_994638_45.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9af775a15a5b82b32259fd5ff63ffd47 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/4fe6a08a-efcb-4785-b3ae-fa3d5c4e5b57_calibrate_994639_53.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/4fe6a08a-efcb-4785-b3ae-fa3d5c4e5b57_calibrate_994639_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:21 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 716 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 790b4dc0144f7a368820780b607d1233 with result from task 716 parsl.dataflow.dflow DEBUG: Task 716 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ca3350e4-2081-4b8b-aad1-cdc8ee1f7aa4_calibrate_994766_17.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/ca3350e4-2081-4b8b-aad1-cdc8ee1f7aa4_calibrate_994766_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:21 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 276b0d807c8ea364dd5248bddb05a3b9 with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/c5511e0c-d81b-4657-8428-e563b9dfd308_calibrate_994766_41.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/c5511e0c-d81b-4657-8428-e563b9dfd308_calibrate_994766_41.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c25c53dc9dca735c3509c2545039dc5b with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/4e47a909-dee6-4e1c-9c70-61a989d3567e_calibrate_994638_48.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/4e47a909-dee6-4e1c-9c70-61a989d3567e_calibrate_994638_48.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b81bd6f7c29362c0b89c52ed7258740 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/24ef6f9c-fa1b-43d9-b531-5f33005eb268_calibrate_994638_14.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/24ef6f9c-fa1b-43d9-b531-5f33005eb268_calibrate_994638_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:21 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:21 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:22 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8706e81b356bd811aa86e58bcad9a2a2 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/942e8574-e6e7-4c7a-a8d3-3a986a3d4cf0_calibrate_994640_34.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/942e8574-e6e7-4c7a-a8d3-3a986a3d4cf0_calibrate_994640_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:22 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e9d4ec37a824432cdaf77b526cd0a0e with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/6b9cdd78-a091-4b70-94ff-75f0d01b9ffd_calibrate_994638_21.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/6b9cdd78-a091-4b70-94ff-75f0d01b9ffd_calibrate_994638_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:22 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:22 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0462510048adf67e20161a214a3d430 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 9c3922d93ca4a74821f29fe8163e3388 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a44a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 1033 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7305e494-32b7-46a6-911c-a23ad1c4bfa3_calibrate_994639_52.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7305e494-32b7-46a6-911c-a23ad1c4bfa3_calibrate_994639_52.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7640ae81-7729-4656-8ec6-bff6d86967ab_characterizeImage_994639_52.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7640ae81-7729-4656-8ec6-bff6d86967ab_characterizeImage_994639_52.stderr parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b602c7f2009894429f7db863910cf609 with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4c35ff2e-a89d-4ac6-a417-944a13d37b71_calibrate_994640_24.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4c35ff2e-a89d-4ac6-a417-944a13d37b71_calibrate_994640_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:22 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:22 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08a5765a5a9eacb3379dcf060cea28bc with result from task 1100 parsl.dataflow.dflow DEBUG: Task 1100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/eb9d520d-15ee-4d38-9635-b7e655e2c6cd_calibrate_994640_36.stdout parsl.dataflow.dflow INFO: Standard error for task 1100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/eb9d520d-15ee-4d38-9635-b7e655e2c6cd_calibrate_994640_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:22 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd5b5cedbc526b246fd1c583a95f67eb with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash 2d3907b5453c4f1b18fcd70b146b6621 parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 1034 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6a64e8c7-2f2b-4b2e-a2e7-01f0b20604d0_calibrate_994639_23.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6a64e8c7-2f2b-4b2e-a2e7-01f0b20604d0_calibrate_994639_23.stderr parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/dc3cc9b3-6ea1-466f-89c0-b191e61b1368_characterizeImage_994639_23.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/dc3cc9b3-6ea1-466f-89c0-b191e61b1368_characterizeImage_994639_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:22 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a5a22f17065cd874b64d5a5100e76620 with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/73ba6db7-cb1b-423c-85a5-223cace7c31b_calibrate_994639_41.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/73ba6db7-cb1b-423c-85a5-223cace7c31b_calibrate_994639_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:23 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1003 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97f4a11ec5f1db8ac2ff0b6b2b424a75 with result from task 1003 parsl.dataflow.dflow DEBUG: Task 1003 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1004 has memoization hash f7798005b7aa5842632ee61764153692 parsl.dataflow.memoization INFO: Task 1004 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1004 try 0 launched on executor multi with executor id 1035 parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/a2889c15-40cc-4fa2-a7d2-bc61b205d4ea_calibrate_994639_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/a2889c15-40cc-4fa2-a7d2-bc61b205d4ea_calibrate_994639_13.stderr parsl.dataflow.dflow INFO: Standard output for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c8b6515e-a8b1-446d-a423-b1175ebea43c_characterizeImage_994639_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1003 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c8b6515e-a8b1-446d-a423-b1175ebea43c_characterizeImage_994639_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:23 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f5821c1af79dae12be9e49a6cb6f2d8 with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d659e998-1007-4a70-9c78-b204de522f17_calibrate_994638_9.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d659e998-1007-4a70-9c78-b204de522f17_calibrate_994638_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:23 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:23 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94f6d9a86799a250417636167c5b75cf with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 17a298fa816b12030b5fb516b823ff23 parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 1036 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/208c1bbd-8549-49a6-bd58-b371620b9197_calibrate_994638_19.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/208c1bbd-8549-49a6-bd58-b371620b9197_calibrate_994638_19.stderr parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a80f6182-59d8-4bae-a8c0-7eb9f04dd738_characterizeImage_994638_19.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a80f6182-59d8-4bae-a8c0-7eb9f04dd738_characterizeImage_994638_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:23 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93d85e4309e2f802b2fbd863005660ff with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/1f6f3c0c-c397-49a1-84c3-b2611c8d3926_calibrate_994639_12.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/1f6f3c0c-c397-49a1-84c3-b2611c8d3926_calibrate_994639_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 143 parsl.dataflow.strategy DEBUG: Executor multi has 143 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 143 parsl.dataflow.strategy DEBUG: Executor multi has 143 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b3587d665e41cefb02a315aecc910fb with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/75c7eac2-ce6e-4a3c-9d04-b6097fdc2e01_calibrate_994766_44.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/75c7eac2-ce6e-4a3c-9d04-b6097fdc2e01_calibrate_994766_44.stderr parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd27b43cf8db86f08024b457a089e067 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b36de761-fd3c-4478-a474-89f49fd6dc36_calibrate_994640_59.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b36de761-fd3c-4478-a474-89f49fd6dc36_calibrate_994640_59.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6c794ecc270a3ef423c9cea0a74284f with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 75027aaede1591f01b34950b5f541c88 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6601080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 1037 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/4d14f8ae-0dc1-4f5a-9f2f-2b507bce68e0_calibrate_994639_9.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/4d14f8ae-0dc1-4f5a-9f2f-2b507bce68e0_calibrate_994639_9.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3db4237a-f4fe-4308-935f-3114402d39da_characterizeImage_994639_9.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3db4237a-f4fe-4308-935f-3114402d39da_characterizeImage_994639_9.stderr parsl.dataflow.dflow INFO: Task 961 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 818a2fa8ed6eced274577f4a27d0e3d0 with result from task 961 parsl.dataflow.dflow DEBUG: Task 961 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 962 has memoization hash 4ef998b35b0b6d23c83d89788cc12ea3 parsl.dataflow.memoization INFO: Task 962 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 962 try 0 launched on executor multi with executor id 1038 parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/d6e14018-61df-416b-bd6d-3cf60152f980_calibrate_994640_38.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/d6e14018-61df-416b-bd6d-3cf60152f980_calibrate_994640_38.stderr parsl.dataflow.dflow INFO: Standard output for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/12b35294-d23b-408e-b524-390548c7ebd2_characterizeImage_994640_38.stdout parsl.dataflow.dflow INFO: Standard error for task 961 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/12b35294-d23b-408e-b524-390548c7ebd2_characterizeImage_994640_38.stderr parsl.dataflow.dflow INFO: Task 932 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74b9b4af72d4da99dc5116e4c746cd9e with result from task 932 parsl.dataflow.dflow DEBUG: Task 932 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/494e52ec-d4d5-476c-a306-f660cd6d6937_calibrate_994765_46.stdout parsl.dataflow.dflow INFO: Standard error for task 932 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/494e52ec-d4d5-476c-a306-f660cd6d6937_calibrate_994765_46.stderr parsl.dataflow.dflow INFO: Task 952 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1a96cccfded86bb86d8acd411525587 with result from task 952 parsl.dataflow.dflow DEBUG: Task 952 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 953 has memoization hash 8753b05f12efcd88ebbcd67ab220b12c parsl.dataflow.memoization INFO: Task 953 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 953 try 0 launched on executor multi with executor id 1039 parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/53923bd6-62c1-4cb6-b31c-e035e29fd015_calibrate_994639_25.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/53923bd6-62c1-4cb6-b31c-e035e29fd015_calibrate_994639_25.stderr parsl.dataflow.dflow INFO: Standard output for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ce7a2248-cb05-4bb8-99e8-9d1a779623a3_characterizeImage_994639_25.stdout parsl.dataflow.dflow INFO: Standard error for task 952 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ce7a2248-cb05-4bb8-99e8-9d1a779623a3_characterizeImage_994639_25.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 140 parsl.dataflow.strategy DEBUG: Executor multi has 140 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d27aa9291e5b40bfe622839d07807db8 with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/02bd8220-8caf-49d7-8c22-9cf453e114da_calibrate_994638_32.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/02bd8220-8caf-49d7-8c22-9cf453e114da_calibrate_994638_32.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9897fb2eb00b40ca9127bbadf8020a1 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/1e0509ec-81d7-4c0d-b2d3-15ce8b5a970b_calibrate_994638_36.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/1e0509ec-81d7-4c0d-b2d3-15ce8b5a970b_calibrate_994638_36.stderr parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c4ddbb33b57aed787c86a8619720047 with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash 65e727066978ecdfcd3ff71a85282df6 parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 1040 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d881b651-c877-4b50-a3f1-9472a492c51c_calibrate_994638_55.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d881b651-c877-4b50-a3f1-9472a492c51c_calibrate_994638_55.stderr parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f5902395-89fe-4e75-96d5-05806be005d3_characterizeImage_994638_55.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f5902395-89fe-4e75-96d5-05806be005d3_characterizeImage_994638_55.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41445637675acf1afe251e828c4dbd0d with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 151f2739f2d3bc4afd062c7c15f2b266 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 1041 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d4cc9b26-b84d-47f6-a0fe-700cee37c926_calibrate_994638_8.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d4cc9b26-b84d-47f6-a0fe-700cee37c926_calibrate_994638_8.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/678b144c-e391-4f17-aff7-92cfd6164e39_characterizeImage_994638_8.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/678b144c-e391-4f17-aff7-92cfd6164e39_characterizeImage_994638_8.stderr parsl.dataflow.dflow INFO: Task 1036 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 035f1d88ad9fde31e3f54ae7baa452e5 with result from task 1036 parsl.dataflow.dflow DEBUG: Task 1036 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1037 has memoization hash 18fd0054a21f988e9c080e236c69d1c7 parsl.dataflow.memoization INFO: Task 1037 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ecfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1037 try 0 launched on executor multi with executor id 1042 parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7f6f2744-d40b-43dc-a58d-a745e4fa685d_calibrate_994639_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7f6f2744-d40b-43dc-a58d-a745e4fa685d_calibrate_994639_19.stderr parsl.dataflow.dflow INFO: Standard output for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/12d5f9d7-57da-491d-a783-06fab4bd441a_characterizeImage_994639_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1036 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/12d5f9d7-57da-491d-a783-06fab4bd441a_characterizeImage_994639_19.stderr parsl.dataflow.dflow INFO: Task 767 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8eb816bb269747b3a073f0aac2f8a14b with result from task 767 parsl.dataflow.dflow DEBUG: Task 767 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/970d08ea-27db-49cf-aa3a-5690296eac32_calibrate_994638_25.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/970d08ea-27db-49cf-aa3a-5690296eac32_calibrate_994638_25.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5df57f077f70b1e948a9979e456322ea with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cb0426c8-affd-4b3b-9355-97485a0fcdff_calibrate_994638_54.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cb0426c8-affd-4b3b-9355-97485a0fcdff_calibrate_994638_54.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f87f6cb3872a69b86ba55bc3440e6eb2 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ca624eae-8f92-4d81-9680-80fe2d870708_calibrate_994638_56.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ca624eae-8f92-4d81-9680-80fe2d870708_calibrate_994638_56.stderr parsl.dataflow.dflow INFO: Task 1061 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b92424a3aab49ece8b6630855b60bd80 with result from task 1061 parsl.dataflow.dflow DEBUG: Task 1061 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/acec8394-f794-4eb5-b80c-47082e1024a2_calibrate_994767_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1061 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/acec8394-f794-4eb5-b80c-47082e1024a2_calibrate_994767_12.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2d985ee4b3d06f12595a2f0f82cdbf2 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 25140ba1c544366cb01175c2bba935f1 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 1043 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/5900c980-53a2-4e2a-8f7e-349b4ecc32d2_calibrate_994638_7.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/5900c980-53a2-4e2a-8f7e-349b4ecc32d2_calibrate_994638_7.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e2c6b254-8c5b-4c5d-af81-80c6b6d71798_characterizeImage_994638_7.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e2c6b254-8c5b-4c5d-af81-80c6b6d71798_characterizeImage_994638_7.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7aa52ab5d7c3a7438651bda641e8e94 with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/71953c32-0db4-47f8-ba43-397b2a724d0e_calibrate_994639_5.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/71953c32-0db4-47f8-ba43-397b2a724d0e_calibrate_994639_5.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d270cdb3a7cf28ef494a35a02726610 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash 088d58d877aff62d07a08387b232e8a6 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 1044 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/0bf85d45-0a37-450f-a86e-a6e9e642225b_calibrate_994639_27.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/0bf85d45-0a37-450f-a86e-a6e9e642225b_calibrate_994639_27.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ed5a4ea9-6a70-4f07-8921-3618d4ae288d_characterizeImage_994639_27.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ed5a4ea9-6a70-4f07-8921-3618d4ae288d_characterizeImage_994639_27.stderr parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeedced385c44dc5eed8144e54e94a4e with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/75ce4930-8a17-4fa0-a137-aa5b92d940db_calibrate_994640_21.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/75ce4930-8a17-4fa0-a137-aa5b92d940db_calibrate_994640_21.stderr parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ca1883ad08748813de1b103a44f5888 with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ab4b0c1e-8415-4c94-83dd-0699436292a5_calibrate_994765_61.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994765/ab4b0c1e-8415-4c94-83dd-0699436292a5_calibrate_994765_61.stderr parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9e4743cfb8eeaa7f88e5484acb0dace with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/a879c77e-cc39-460f-ab77-2e34d27573e5_calibrate_994766_12.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/a879c77e-cc39-460f-ab77-2e34d27573e5_calibrate_994766_12.stderr parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1bc025d1733de287ec789f5a7290b2a with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/db218131-767e-40d0-afa0-22f45d7a0d93_calibrate_994767_13.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/db218131-767e-40d0-afa0-22f45d7a0d93_calibrate_994767_13.stderr parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed1149ec2bdb1e9b6fb72cc028461aad with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c0554aaf-7434-4e1f-b21a-afa051f9ee33_calibrate_994639_58.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c0554aaf-7434-4e1f-b21a-afa051f9ee33_calibrate_994639_58.stderr parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0fe93b086769d4f1caf2ff7f5df991a with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash d842c9929d0638b61338a7c21ab44b93 parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 1045 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/06db42e4-757a-427f-bb56-288d01f484b7_calibrate_994640_51.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/06db42e4-757a-427f-bb56-288d01f484b7_calibrate_994640_51.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c8c1bc09-c19e-4303-933c-d2742446db55_characterizeImage_994640_51.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c8c1bc09-c19e-4303-933c-d2742446db55_characterizeImage_994640_51.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b803ed2bba63295e52470c31c34ad75 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/41ea3304-9116-41cf-bc8b-3c11c18f4ce8_calibrate_994639_11.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/41ea3304-9116-41cf-bc8b-3c11c18f4ce8_calibrate_994639_11.stderr parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f54ecf7505c2bbb769d3f1b8731f665 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/af08614a-4ba9-419d-b53c-fd969fc981e6_calibrate_994639_26.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/af08614a-4ba9-419d-b53c-fd969fc981e6_calibrate_994639_26.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 126 parsl.dataflow.strategy DEBUG: Executor multi has 126 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 911 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f23a3fd2d7fad03f590ee281721b0bf2 with result from task 911 parsl.dataflow.dflow DEBUG: Task 911 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4fa03677-51ff-40d9-980b-c5a529d426a9_calibrate_994640_60.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4fa03677-51ff-40d9-980b-c5a529d426a9_calibrate_994640_60.stderr parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 078a2964c737eb2063116ac5659ac6fb with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/dfa8f0e6-665d-4615-a2d0-96c8d45d84db_calibrate_994640_35.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/dfa8f0e6-665d-4615-a2d0-96c8d45d84db_calibrate_994640_35.stderr parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3efa2c0bd1a7f0b43d6a4007e79b6e3c with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 98d4bbad57b5e039e734d9c21bfec797 parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64851c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 1046 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/fc26323c-fe4b-458b-9e85-dd0cd80d01c8_calibrate_994638_34.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/fc26323c-fe4b-458b-9e85-dd0cd80d01c8_calibrate_994638_34.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/2d156ecd-7970-4580-a3cd-264de6b6e548_characterizeImage_994638_34.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/2d156ecd-7970-4580-a3cd-264de6b6e548_characterizeImage_994638_34.stderr parsl.dataflow.dflow INFO: Task 971 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47e5cc4e7e8d6a619fc585e19a964db0 with result from task 971 parsl.dataflow.dflow DEBUG: Task 971 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ee43ae39-14b0-46c7-a53e-b6fe1e2e5950_calibrate_994638_46.stdout parsl.dataflow.dflow INFO: Standard error for task 971 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ee43ae39-14b0-46c7-a53e-b6fe1e2e5950_calibrate_994638_46.stderr parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40a1e82209c4becb5aace0c3159a9e61 with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/7a8f5a7d-5a10-4dc4-a1f6-c369278d879e_calibrate_994640_6.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/7a8f5a7d-5a10-4dc4-a1f6-c369278d879e_calibrate_994640_6.stderr parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed34ce52c4b80bb4e793cafc692ef2f1 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash a3d6e77b21a2c7fd3f2c3902d6fd83ce parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 1047 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/13decf0b-2b11-4031-8865-9650fb3fbda6_calibrate_994640_22.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/13decf0b-2b11-4031-8865-9650fb3fbda6_calibrate_994640_22.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b76576dd-f549-45ba-b4c9-116ee2956270_characterizeImage_994640_22.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b76576dd-f549-45ba-b4c9-116ee2956270_characterizeImage_994640_22.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee44b6037f851c6af4f4e9dc518f5f89 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/0c7e830e-e5b3-40c4-aa62-77425cea9cb7_calibrate_994639_3.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/0c7e830e-e5b3-40c4-aa62-77425cea9cb7_calibrate_994639_3.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 286e16cd60af0a92da8aa33911472cbd with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 810233a47ca08f84d049125018525fa0 parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e7d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 1048 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f0b2c60c-742a-45e4-8fa2-1b605c37e17e_calibrate_994640_53.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f0b2c60c-742a-45e4-8fa2-1b605c37e17e_calibrate_994640_53.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/91fc2fde-3241-400e-a1b4-65e4d8e75f4a_characterizeImage_994640_53.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/91fc2fde-3241-400e-a1b4-65e4d8e75f4a_characterizeImage_994640_53.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1eb18211defb2d203803cd178a18dc7 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/d20a2dfe-f096-47ca-b87d-099b56ea7892_calibrate_994639_51.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/d20a2dfe-f096-47ca-b87d-099b56ea7892_calibrate_994639_51.stderr parsl.dataflow.dflow INFO: Task 941 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7602e316443e5c5ccfc73cd96573596c with result from task 941 parsl.dataflow.dflow DEBUG: Task 941 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/aaefea23-455c-4937-a854-3ef527929201_calibrate_994639_55.stdout parsl.dataflow.dflow INFO: Standard error for task 941 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/aaefea23-455c-4937-a854-3ef527929201_calibrate_994639_55.stderr parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a61e8ca349796cdeaea2dbb4b2ff7cb with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f10e2cac-e075-42cb-8dba-5b9d350a6e84_calibrate_994766_38.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/f10e2cac-e075-42cb-8dba-5b9d350a6e84_calibrate_994766_38.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 118 parsl.dataflow.strategy DEBUG: Executor multi has 118 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 1084 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01c82e7391d5197b1db3bd1af05ebfc4 with result from task 1084 parsl.dataflow.dflow DEBUG: Task 1084 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1085 has memoization hash 23a05d1a0eb1b4c54161e4f23d5687f7 parsl.dataflow.memoization INFO: Task 1085 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62efe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1085 try 0 launched on executor multi with executor id 1049 parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/615f5161-2e0a-46f5-b2ff-fff7c069529f_calibrate_994640_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/615f5161-2e0a-46f5-b2ff-fff7c069529f_calibrate_994640_26.stderr parsl.dataflow.dflow INFO: Standard output for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/95de70d5-4289-4d30-968a-e81e640d9371_characterizeImage_994640_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1084 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/95de70d5-4289-4d30-968a-e81e640d9371_characterizeImage_994640_26.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4459bf39bf69918db5297c96c074dd2a with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/39bcbb88-ecef-4774-8d20-02faec0b6925_calibrate_994638_10.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/39bcbb88-ecef-4774-8d20-02faec0b6925_calibrate_994638_10.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89a1e997e8d55c1e3d1ad9ffcec7fe0e with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c3fe9734-a55d-40f2-a492-3fc8e7521164_calibrate_994638_3.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c3fe9734-a55d-40f2-a492-3fc8e7521164_calibrate_994638_3.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83bb2494bdb9fb83b5a4aa61e672f6db with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash f07cee7d824749ecea3d0e3e67ae4508 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e5e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 1050 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/89d81d5f-d166-4fe8-8729-c8e3c1696e67_calibrate_994640_43.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/89d81d5f-d166-4fe8-8729-c8e3c1696e67_calibrate_994640_43.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c7310e47-392a-4f8c-b329-c2a120a67a33_characterizeImage_994640_43.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c7310e47-392a-4f8c-b329-c2a120a67a33_characterizeImage_994640_43.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00b7c70247dd50cc52372ae2dcc5b09c with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 49086b4ec4c4220632027a6146107e61 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 1051 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/dc90d615-7a91-4fee-ac58-373e442d2fb9_calibrate_994639_36.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/dc90d615-7a91-4fee-ac58-373e442d2fb9_calibrate_994639_36.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3db63ef5-7b61-49c4-a4d1-75af970dba37_characterizeImage_994639_36.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3db63ef5-7b61-49c4-a4d1-75af970dba37_characterizeImage_994639_36.stderr parsl.dataflow.dflow INFO: Task 1000 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5166306907906c0087acdfa77312be5a with result from task 1000 parsl.dataflow.dflow DEBUG: Task 1000 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1001 has memoization hash c5e5f3c07959e0d944327d758f939c23 parsl.dataflow.memoization INFO: Task 1001 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a6340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1001 try 0 launched on executor multi with executor id 1052 parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2262daf0-4102-4900-a5db-b3ab1c790233_calibrate_994640_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2262daf0-4102-4900-a5db-b3ab1c790233_calibrate_994640_10.stderr parsl.dataflow.dflow INFO: Standard output for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0ffedbe8-2667-47f0-905c-610fa51e178e_characterizeImage_994640_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1000 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0ffedbe8-2667-47f0-905c-610fa51e178e_characterizeImage_994640_10.stderr parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f80ff5e61a37299517616f408bec994 with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/82f83db5-e03c-46d5-92d4-d6e14c7574ea_calibrate_994639_10.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/82f83db5-e03c-46d5-92d4-d6e14c7574ea_calibrate_994639_10.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 805966df981e518620ead2f90f60dd5d with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7949719f-15f3-4ae0-98e8-5947bb4e74df_calibrate_994767_49.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/7949719f-15f3-4ae0-98e8-5947bb4e74df_calibrate_994767_49.stderr parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf7ac87cde2d337cad684582d4b28cc4 with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9e7a7155-8669-4c5b-9ceb-a87141fded12_calibrate_994639_40.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9e7a7155-8669-4c5b-9ceb-a87141fded12_calibrate_994639_40.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67079c893ae4d9905ab1886286f79a69 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/40a66c23-1c65-4361-bd3c-c06600a2bde4_calibrate_994639_59.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/40a66c23-1c65-4361-bd3c-c06600a2bde4_calibrate_994639_59.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6adc6e687ffbe15fea7cf3a144adec64 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b8d62ae9-8553-485a-b82b-438fb77a64f1_calibrate_994640_8.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b8d62ae9-8553-485a-b82b-438fb77a64f1_calibrate_994640_8.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8641190fe0c3076b85d0ccb79ac3b3fc with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash 707c8ef7648d9efd443b39b4bebf63e7 parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 1053 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/db919ba6-fe1b-4985-a633-214a47d536b3_calibrate_994638_47.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/db919ba6-fe1b-4985-a633-214a47d536b3_calibrate_994638_47.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f165efcf-3589-4119-b9f7-66a3696e704b_characterizeImage_994638_47.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f165efcf-3589-4119-b9f7-66a3696e704b_characterizeImage_994638_47.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 743aab7ff0322b5fccf7a14baa4fdfd8 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/eab36fd5-92d3-4b02-9b77-14878a58702d_calibrate_994640_62.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/eab36fd5-92d3-4b02-9b77-14878a58702d_calibrate_994640_62.stderr parsl.dataflow.dflow INFO: Task 967 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37c8081e6cb97e7b93247ebb58cddc00 with result from task 967 parsl.dataflow.dflow DEBUG: Task 967 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 968 has memoization hash 69c114444c33486307f7fa1e5db1e810 parsl.dataflow.memoization INFO: Task 968 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a4680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 968 try 0 launched on executor multi with executor id 1054 parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c424725c-cd7f-460b-9e08-2f5d5c287e48_calibrate_994639_43.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c424725c-cd7f-460b-9e08-2f5d5c287e48_calibrate_994639_43.stderr parsl.dataflow.dflow INFO: Standard output for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f66e86fa-b6cd-49ab-a5a4-c80153f02934_characterizeImage_994639_43.stdout parsl.dataflow.dflow INFO: Standard error for task 967 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/f66e86fa-b6cd-49ab-a5a4-c80153f02934_characterizeImage_994639_43.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 862d41701dc0513b3cf92d45b02b35f9 with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8ab050be-96c3-45ae-bb17-4b02f9751fca_calibrate_994639_20.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8ab050be-96c3-45ae-bb17-4b02f9751fca_calibrate_994639_20.stderr parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92ac4dea511f064b8da938a72f9ed132 with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/516ee58b-bd2e-436b-be97-f5a6dafaf032_calibrate_994640_42.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/516ee58b-bd2e-436b-be97-f5a6dafaf032_calibrate_994640_42.stderr parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b164c5171e98a1c0e912ee2014044517 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash 831c86be99ca97107e1fd5453383198d parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6485b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 1055 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ff1baab4-cf1d-49ae-8046-bcb4223b3565_calibrate_994638_39.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ff1baab4-cf1d-49ae-8046-bcb4223b3565_calibrate_994638_39.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b20c8eb3-d565-4a3d-bf37-06ca2436cec5_characterizeImage_994638_39.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/b20c8eb3-d565-4a3d-bf37-06ca2436cec5_characterizeImage_994638_39.stderr parsl.dataflow.dflow INFO: Task 1019 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cddc575cb0f5a86ee23714ea65b329c1 with result from task 1019 parsl.dataflow.dflow DEBUG: Task 1019 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7ad13248-2307-404d-a128-8b505b129729_calibrate_994639_29.stdout parsl.dataflow.dflow INFO: Standard error for task 1019 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7ad13248-2307-404d-a128-8b505b129729_calibrate_994639_29.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54965e975a8ef08a3512890386ee92bf with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2fa2fab1-3ff1-4532-92bb-a9ae7ee84a45_calibrate_994640_41.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2fa2fab1-3ff1-4532-92bb-a9ae7ee84a45_calibrate_994640_41.stderr parsl.dataflow.dflow INFO: Task 1090 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87d456b582a27753dcffef3c1411e288 with result from task 1090 parsl.dataflow.dflow DEBUG: Task 1090 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1091 has memoization hash 6c1767d84d3171f700beda3375b4a383 parsl.dataflow.memoization INFO: Task 1091 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1091 try 0 launched on executor multi with executor id 1056 parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/9fd165b9-321b-4fbb-b5ee-d42f5aef50ba_calibrate_994640_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/9fd165b9-321b-4fbb-b5ee-d42f5aef50ba_calibrate_994640_12.stderr parsl.dataflow.dflow INFO: Standard output for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/56411ec4-2621-4f17-a373-cecce63eec39_characterizeImage_994640_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1090 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/56411ec4-2621-4f17-a373-cecce63eec39_characterizeImage_994640_12.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbe614cecc42fa295a78a520eef82fe3 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash f4843c380ebc0897528a5f4c546c8212 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 1057 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/04f6f649-71be-4aa2-a31f-c6dc110ce37a_calibrate_994640_20.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/04f6f649-71be-4aa2-a31f-c6dc110ce37a_calibrate_994640_20.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0145a0e7-051d-4e00-a728-0fcfa04f55d6_characterizeImage_994640_20.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/0145a0e7-051d-4e00-a728-0fcfa04f55d6_characterizeImage_994640_20.stderr parsl.dataflow.dflow INFO: Task 1057 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55dd99cc5c70d3c44dadab49d25014d0 with result from task 1057 parsl.dataflow.dflow DEBUG: Task 1057 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1058 has memoization hash 2d1ff7dcd577784cd7f4bd7f2e240201 parsl.dataflow.memoization INFO: Task 1058 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ee5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1058 try 0 launched on executor multi with executor id 1058 parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/f4ebd377-2325-4703-9ee7-b7abc6661f09_calibrate_994639_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/f4ebd377-2325-4703-9ee7-b7abc6661f09_calibrate_994639_42.stderr parsl.dataflow.dflow INFO: Standard output for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/193590eb-490a-423c-827a-b4f57d1be6e6_characterizeImage_994639_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1057 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/193590eb-490a-423c-827a-b4f57d1be6e6_characterizeImage_994639_42.stderr parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4576f01870643b04220cc8a53f788603 with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash d2b865f98fb9e203660b5a9d59ca2c53 parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-15 04:16:43 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 1059 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8d0916f8-fb3a-48fd-9cad-e61b0d79d3bf_calibrate_994639_30.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8d0916f8-fb3a-48fd-9cad-e61b0d79d3bf_calibrate_994639_30.stderr parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b674087d-4466-4bd1-82ca-d267bcc9f71e_characterizeImage_994639_30.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/b674087d-4466-4bd1-82ca-d267bcc9f71e_characterizeImage_994639_30.stderr 2024-03-15 04:16:43 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b3458b57121052066a7aa2d22f7a6d4 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/db43b2c4-e553-4220-a967-276a9afbf10c_calibrate_994640_9.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/db43b2c4-e553-4220-a967-276a9afbf10c_calibrate_994640_9.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1a324f5b5dfded4c0b8b635b75981ab with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/64ca9717-a5e6-48ec-94fd-bd04196b6423_calibrate_994638_17.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/64ca9717-a5e6-48ec-94fd-bd04196b6423_calibrate_994638_17.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 036a162e83600cf0d5b730611a8a3abc with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash e70d82f032581c1c49633667541f8b04 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a47c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 1060 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c8330652-9fb2-4681-a0f1-180275601da1_calibrate_994639_47.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c8330652-9fb2-4681-a0f1-180275601da1_calibrate_994639_47.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5445d472-e3bb-4ee6-b3fb-2c77934ae913_characterizeImage_994639_47.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5445d472-e3bb-4ee6-b3fb-2c77934ae913_characterizeImage_994639_47.stderr parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee7034d15509c666ac7567c3044d339c with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash 8dcfb1dc8326d9d7b3fab3f0ee3f0c10 parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 1061 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/baf5ff65-22b9-4f53-a3bb-0ade4cf21f29_calibrate_994640_45.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/baf5ff65-22b9-4f53-a3bb-0ade4cf21f29_calibrate_994640_45.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/fecbaa73-e12d-4c85-921c-5c356a0308b9_characterizeImage_994640_45.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/fecbaa73-e12d-4c85-921c-5c356a0308b9_characterizeImage_994640_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:45 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:46 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 527bb518657b74bae4f7c4624a7992c1 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 584161fc39e7d95640004e2b9c479fc8 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65a79c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 1062 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/1c1e538a-4709-4e5b-b173-d5e4ff99fe69_calibrate_994639_28.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/1c1e538a-4709-4e5b-b173-d5e4ff99fe69_calibrate_994639_28.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7b7eda5e-521a-45ae-8876-eb78c477b726_characterizeImage_994639_28.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/7b7eda5e-521a-45ae-8876-eb78c477b726_characterizeImage_994639_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:47 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be33e4e577ce87f215d84b3d2ccee174 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/15f4b262-4f2f-405e-a08d-a276953a924d_calibrate_994639_57.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/15f4b262-4f2f-405e-a08d-a276953a924d_calibrate_994639_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:47 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:47 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1013 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b03e2e92a2e17004ec4d5125d41de14f with result from task 1013 parsl.dataflow.dflow DEBUG: Task 1013 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/341363f1-50ce-4a08-afdf-bf6610284e9d_calibrate_994639_4.stdout parsl.dataflow.dflow INFO: Standard error for task 1013 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/341363f1-50ce-4a08-afdf-bf6610284e9d_calibrate_994639_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:48 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:48 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:49 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84adfd2e18e8f91dea51693108a9d49e with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/fc4434dc-37ae-4951-984b-d3a5477a0e55_calibrate_994640_16.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/fc4434dc-37ae-4951-984b-d3a5477a0e55_calibrate_994640_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:49 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c31d6f6c8664f073ed19ab863ff13b0d with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4b7a741d-e5c7-4ffc-b5fa-bfec48c6880b_calibrate_994640_37.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4b7a741d-e5c7-4ffc-b5fa-bfec48c6880b_calibrate_994640_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:49 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1871bfdea56de215cc2a0f771c1c1f57 with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 158f3a7c5147eab64420853020ba9432 parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 1063 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/0a86d611-efee-4f1a-b332-83a0b7459913_calibrate_994640_50.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/0a86d611-efee-4f1a-b332-83a0b7459913_calibrate_994640_50.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2a357535-cd81-4e24-9d73-e7ef047439ba_characterizeImage_994640_50.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2a357535-cd81-4e24-9d73-e7ef047439ba_characterizeImage_994640_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:50 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 039d5f2ae257f21f68ab622082ebbba4 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4905f9b4-22bb-45a5-92a0-86b5f39b3697_calibrate_994640_58.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4905f9b4-22bb-45a5-92a0-86b5f39b3697_calibrate_994640_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:50 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:50 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:51 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8bef757e2eb244b33f4c32b43e5f615 with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/02662a13-c59d-4982-af22-38581b4117c9_calibrate_994638_20.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/02662a13-c59d-4982-af22-38581b4117c9_calibrate_994638_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:51 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:51 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d5450e139d6edc046b57c76776ece76 with result from task 802 parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 3fe7b6bf2417532be935b93af37f7f12 parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 1064 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cc322e09-580f-4933-b020-59d11f6086de_calibrate_994638_49.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cc322e09-580f-4933-b020-59d11f6086de_calibrate_994638_49.stderr parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/63017076-4c6a-4833-a6e8-975967ff6587_characterizeImage_994638_49.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/63017076-4c6a-4833-a6e8-975967ff6587_characterizeImage_994638_49.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 356a5c04a9f67b7594e94cc6f222066d with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 586bcd792a80bde2a30689c56c3aaf32 parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 1065 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/3b24fb30-ef49-43f2-bd14-265d8b8d36c2_calibrate_994639_48.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/3b24fb30-ef49-43f2-bd14-265d8b8d36c2_calibrate_994639_48.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3c1a8bc2-d505-4cb0-a024-4ddff488798e_characterizeImage_994639_48.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/3c1a8bc2-d505-4cb0-a024-4ddff488798e_characterizeImage_994639_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:52 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d27b9007f1152eb7dc633d41ba0709df with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/62e289a2-fa3c-42ee-b6ed-3f4c58079e95_calibrate_994639_21.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/62e289a2-fa3c-42ee-b6ed-3f4c58079e95_calibrate_994639_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:52 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a840611731334cf0a2ee1903f623115 with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6c65ff0d-301c-4b2e-b057-fb20c39626eb_calibrate_994639_1.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6c65ff0d-301c-4b2e-b057-fb20c39626eb_calibrate_994639_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:53 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:53 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e977c61dfb118c4ddae251aeaf690fd with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash 05979316784ad3ce65a26d8a3f89eb7f parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 1066 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/3d1118d6-a38f-45a9-a592-18a7d9d586c4_calibrate_994639_60.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/3d1118d6-a38f-45a9-a592-18a7d9d586c4_calibrate_994639_60.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5e03f5da-0c50-4734-bfca-3cf6f3deca50_characterizeImage_994639_60.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5e03f5da-0c50-4734-bfca-3cf6f3deca50_characterizeImage_994639_60.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d95ae7f930e3f11a8585963741c4571 with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash 65dfbd5ff6cc17f1201a0129ae4bff64 parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 1067 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/2de62aea-9188-4b20-aa50-3b54f74abc56_calibrate_994638_57.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/2de62aea-9188-4b20-aa50-3b54f74abc56_calibrate_994638_57.stderr parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/242a0e9f-12c4-4563-a445-895b63195a2d_characterizeImage_994638_57.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/242a0e9f-12c4-4563-a445-895b63195a2d_characterizeImage_994638_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:54 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:54 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:55 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:55 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 869eba2dea750735b50a0d16afd3e454 with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash 3b9a2004a715edc9b1a7a43110ed9fc6 parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63928e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 1068 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/6300c284-eb8f-49c5-900b-c3d74a21c638_calibrate_994638_13.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/6300c284-eb8f-49c5-900b-c3d74a21c638_calibrate_994638_13.stderr parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9e98434c-ca64-44ef-bb5e-634632f81af9_characterizeImage_994638_13.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9e98434c-ca64-44ef-bb5e-634632f81af9_characterizeImage_994638_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:56 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:56 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 471464f17e6b86082cd5a01940afae9e with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/0b4c133b-78c9-483d-8d7e-6e3e038d8eb8_calibrate_994638_6.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/0b4c133b-78c9-483d-8d7e-6e3e038d8eb8_calibrate_994638_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:57 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:57 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed92dc7422db111c5de891cbbc2bf381 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 5fcb18eabc190f34339f433c2a667678 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 1069 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b20c70e0-f04f-4ad9-8501-ee543054fd49_calibrate_994640_14.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b20c70e0-f04f-4ad9-8501-ee543054fd49_calibrate_994640_14.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/43a765cf-cb37-41a9-8a8c-55209d01851c_characterizeImage_994640_14.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/43a765cf-cb37-41a9-8a8c-55209d01851c_characterizeImage_994640_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:58 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:58 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f1cc87624f4fe97616f01ba10ba8dcf with result from task 524 parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:59 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/e05cd477-65b1-4301-ae31-94f6070f3fc7_calibrate_994640_17.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/e05cd477-65b1-4301-ae31-94f6070f3fc7_calibrate_994640_17.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d975d1d2c27aeede90a2ec9b1093dac with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 32d5a0ebfaf0015d1a9d5c067f6b62f0 parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63de8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 1070 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/04c905fa-e8ef-4a57-a4ee-16f15b711b8c_calibrate_994638_52.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/04c905fa-e8ef-4a57-a4ee-16f15b711b8c_calibrate_994638_52.stderr parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e6ed663b-11f6-4971-a214-53bafda1b55a_characterizeImage_994638_52.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/e6ed663b-11f6-4971-a214-53bafda1b55a_characterizeImage_994638_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:16:59 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5ee838b48e02787e41ad586bb061524 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 72d00df9c2c0b8ca3fb7315f23a47ea1 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 1071 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5b07b8cc-af5b-45f9-a55a-8a2c5232f2c6_calibrate_994640_13.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5b07b8cc-af5b-45f9-a55a-8a2c5232f2c6_calibrate_994640_13.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2b9dd577-a70f-449d-ac96-98abf4b1ffbb_characterizeImage_994640_13.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2b9dd577-a70f-449d-ac96-98abf4b1ffbb_characterizeImage_994640_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:00 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5df61b240fdb99c19f30ba7127048a0e with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/51f4b98e-c867-4d29-9a6e-a117e258e5d9_calibrate_994640_3.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/51f4b98e-c867-4d29-9a6e-a117e258e5d9_calibrate_994640_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:00 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51d26baa3691bd8554a4f2daf5ea4dd5 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e51fae17-5c80-41b9-9b15-7608c24a7091_calibrate_994638_27.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e51fae17-5c80-41b9-9b15-7608c24a7091_calibrate_994638_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:00 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 962 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ef998b35b0b6d23c83d89788cc12ea3 with result from task 962 parsl.dataflow.dflow DEBUG: Task 962 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/d6e14018-61df-416b-bd6d-3cf60152f980_calibrate_994640_38.stdout parsl.dataflow.dflow INFO: Standard error for task 962 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/d6e14018-61df-416b-bd6d-3cf60152f980_calibrate_994640_38.stderr parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d3907b5453c4f1b18fcd70b146b6621 with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6a64e8c7-2f2b-4b2e-a2e7-01f0b20604d0_calibrate_994639_23.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/6a64e8c7-2f2b-4b2e-a2e7-01f0b20604d0_calibrate_994639_23.stderr parsl.dataflow.dflow INFO: Task 1064 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fd802daa686375f09c91de5385123ef with result from task 1064 parsl.dataflow.dflow DEBUG: Task 1064 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c85f8af1-3279-4c5b-ab76-74a61c0e045c_calibrate_994638_35.stdout parsl.dataflow.dflow INFO: Standard error for task 1064 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c85f8af1-3279-4c5b-ab76-74a61c0e045c_calibrate_994638_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:01 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c3922d93ca4a74821f29fe8163e3388 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7305e494-32b7-46a6-911c-a23ad1c4bfa3_calibrate_994639_52.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7305e494-32b7-46a6-911c-a23ad1c4bfa3_calibrate_994639_52.stderr parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17a298fa816b12030b5fb516b823ff23 with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/208c1bbd-8549-49a6-bd58-b371620b9197_calibrate_994638_19.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/208c1bbd-8549-49a6-bd58-b371620b9197_calibrate_994638_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:01 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 166a77cbec5d8b955782f97d4ca84780 with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 8d10b0cf4382757e73f67befa0ff0fa6 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6603600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 1072 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/fa0cdc5c-3502-437a-84ee-fbba6af59680_calibrate_994639_34.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/fa0cdc5c-3502-437a-84ee-fbba6af59680_calibrate_994639_34.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8f10ce28-85a0-46d7-aafe-f989c3089c47_characterizeImage_994639_34.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/8f10ce28-85a0-46d7-aafe-f989c3089c47_characterizeImage_994639_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:02 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97b0512550fa1589cfdffc2ab06de4c5 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/e4db660d-c598-4daf-bf51-46e8f185488f_calibrate_994639_8.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/e4db660d-c598-4daf-bf51-46e8f185488f_calibrate_994639_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:02 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:03 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 953 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8753b05f12efcd88ebbcd67ab220b12c with result from task 953 parsl.dataflow.dflow DEBUG: Task 953 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/53923bd6-62c1-4cb6-b31c-e035e29fd015_calibrate_994639_25.stdout parsl.dataflow.dflow INFO: Standard error for task 953 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/53923bd6-62c1-4cb6-b31c-e035e29fd015_calibrate_994639_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:03 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c21c7e83ba881c4fb228e290fe7c8a21 with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/b75b0834-6c42-4fc1-88ea-93181bfc04b3_calibrate_994638_45.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/b75b0834-6c42-4fc1-88ea-93181bfc04b3_calibrate_994638_45.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ccf7c389926fa938ab019d9b7917107 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/371a7754-2619-485d-a094-9a536d461f99_calibrate_994639_37.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/371a7754-2619-485d-a094-9a536d461f99_calibrate_994639_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:04 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:04 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:05 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:05 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 1051 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44ac161be677512574f62ca22d3f4573 with result from task 1051 parsl.dataflow.dflow DEBUG: Task 1051 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1052 has memoization hash 481f9f7981ee1e1c861d9eaa276e2388 parsl.dataflow.memoization INFO: Task 1052 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62edf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1052 try 0 launched on executor multi with executor id 1073 parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/074aa2f0-a1f0-4f0d-866e-b569fca48158_calibrate_994639_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/074aa2f0-a1f0-4f0d-866e-b569fca48158_calibrate_994639_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5a493726-ba9b-4f66-ac60-e3fc283ee052_characterizeImage_994639_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1051 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/5a493726-ba9b-4f66-ac60-e3fc283ee052_characterizeImage_994639_61.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75027aaede1591f01b34950b5f541c88 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/4d14f8ae-0dc1-4f5a-9f2f-2b507bce68e0_calibrate_994639_9.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/4d14f8ae-0dc1-4f5a-9f2f-2b507bce68e0_calibrate_994639_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:06 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 1004 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7798005b7aa5842632ee61764153692 with result from task 1004 parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:07 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow DEBUG: Task 1004 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/a2889c15-40cc-4fa2-a7d2-bc61b205d4ea_calibrate_994639_13.stdout parsl.dataflow.dflow INFO: Standard error for task 1004 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/a2889c15-40cc-4fa2-a7d2-bc61b205d4ea_calibrate_994639_13.stderr parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87105170aec62fd2ec3c77bdf35c805a with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash de7bfaee788a820376693115a8107f3a parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 1074 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/34b27ed5-0ede-4824-8609-e8b9aa620089_calibrate_994640_1.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/34b27ed5-0ede-4824-8609-e8b9aa620089_calibrate_994640_1.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/4995ebc0-1b0c-4771-8a05-1c1f20ecb7e3_characterizeImage_994640_1.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/4995ebc0-1b0c-4771-8a05-1c1f20ecb7e3_characterizeImage_994640_1.stderr parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d842c9929d0638b61338a7c21ab44b93 with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/06db42e4-757a-427f-bb56-288d01f484b7_calibrate_994640_51.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/06db42e4-757a-427f-bb56-288d01f484b7_calibrate_994640_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:07 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:08 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 1037 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18fd0054a21f988e9c080e236c69d1c7 with result from task 1037 parsl.dataflow.dflow DEBUG: Task 1037 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7f6f2744-d40b-43dc-a58d-a745e4fa685d_calibrate_994639_19.stdout parsl.dataflow.dflow INFO: Standard error for task 1037 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7f6f2744-d40b-43dc-a58d-a745e4fa685d_calibrate_994639_19.stderr parsl.dataflow.dflow INFO: Task 995 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ff00a58790a2f046a1ccc5a6dc88142 with result from task 995 parsl.dataflow.dflow DEBUG: Task 995 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/f5b1855e-1e27-472b-a912-8be71cad9fd7_calibrate_994638_51.stdout parsl.dataflow.dflow INFO: Standard error for task 995 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/f5b1855e-1e27-472b-a912-8be71cad9fd7_calibrate_994638_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:08 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:09 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:09 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29af2a519024b3c907f46d0ce6cb1014 with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash f23dd614621e24bb65b154e954dc50de parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66525c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 1075 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4a8edad1-a023-4abb-b143-36acce90188f_calibrate_994640_57.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4a8edad1-a023-4abb-b143-36acce90188f_calibrate_994640_57.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e1ac32e5-9b56-41b5-a37c-bf41c25e39ba_characterizeImage_994640_57.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/e1ac32e5-9b56-41b5-a37c-bf41c25e39ba_characterizeImage_994640_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:09 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25140ba1c544366cb01175c2bba935f1 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/5900c980-53a2-4e2a-8f7e-349b4ecc32d2_calibrate_994638_7.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/5900c980-53a2-4e2a-8f7e-349b4ecc32d2_calibrate_994638_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:10 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:10 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:11 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:11 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49e7dfcd367a1f754e04b01e9719fce3 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash e45b17cae165986dea131b12cbb1e398 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 1076 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c7d20f97-2e3e-468a-8fbc-dd09aad82566_calibrate_994639_32.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c7d20f97-2e3e-468a-8fbc-dd09aad82566_calibrate_994639_32.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ff16fbcd-e446-4b1f-a1d4-60dfe78232ad_characterizeImage_994639_32.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/ff16fbcd-e446-4b1f-a1d4-60dfe78232ad_characterizeImage_994639_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:11 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:12 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc4d43ec41c31144e50a0b083eccf0f7 with result from task 1114 parsl.dataflow.dflow DEBUG: Task 1114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1115 has memoization hash 51fc895f1af2578c81fe69bbaf5a2674 parsl.dataflow.memoization INFO: Task 1115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6346160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1115 try 0 launched on executor multi with executor id 1077 parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/acd388ff-8323-4d58-b286-354294dfcdef_calibrate_994640_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/acd388ff-8323-4d58-b286-354294dfcdef_calibrate_994640_47.stderr parsl.dataflow.dflow INFO: Standard output for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5a2ac9b2-36cf-4815-aa5a-91b01a773eea_characterizeImage_994640_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/5a2ac9b2-36cf-4815-aa5a-91b01a773eea_characterizeImage_994640_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:12 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 088d58d877aff62d07a08387b232e8a6 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/0bf85d45-0a37-450f-a86e-a6e9e642225b_calibrate_994639_27.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/0bf85d45-0a37-450f-a86e-a6e9e642225b_calibrate_994639_27.stderr parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b68f5d7e8c2f60bfd30883997d69c45 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash ffdb35197f3edb2f80524a24468eaf05 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 1078 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/89e6d182-14ac-438d-9606-999cbf339a4c_calibrate_994638_5.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/89e6d182-14ac-438d-9606-999cbf339a4c_calibrate_994638_5.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/5094fd64-88b9-4d8c-8adc-e39a8b843616_characterizeImage_994638_5.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/5094fd64-88b9-4d8c-8adc-e39a8b843616_characterizeImage_994638_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:13 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:13 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 985 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc00f28e4eaaab51346e6c2c8f8321fb with result from task 985 parsl.dataflow.dflow DEBUG: Task 985 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 986 has memoization hash b15e1f76a15a9e173fcc52d6e52e4d4a parsl.dataflow.memoization INFO: Task 986 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62a5620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 986 try 0 launched on executor multi with executor id 1079 parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/1836d583-e490-40b1-96b8-1f4c96908f4d_calibrate_994640_11.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/1836d583-e490-40b1-96b8-1f4c96908f4d_calibrate_994640_11.stderr parsl.dataflow.dflow INFO: Standard output for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/a435d552-5623-482b-b7b6-d30bbb8e8246_characterizeImage_994640_11.stdout parsl.dataflow.dflow INFO: Standard error for task 985 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/a435d552-5623-482b-b7b6-d30bbb8e8246_characterizeImage_994640_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:13 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa5e34874f2a70b714a9eedf13581a01 with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash acd9d325f8294d3ee8993f1c1b84768b parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 1080 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/fe563bff-de37-4f03-a1af-69fe31687a4a_calibrate_994639_38.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/fe563bff-de37-4f03-a1af-69fe31687a4a_calibrate_994639_38.stderr parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c854eb81-4cf9-4185-9e99-9f519ca8ef43_characterizeImage_994639_38.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/c854eb81-4cf9-4185-9e99-9f519ca8ef43_characterizeImage_994639_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:14 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3d6e77b21a2c7fd3f2c3902d6fd83ce with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/13decf0b-2b11-4031-8865-9650fb3fbda6_calibrate_994640_22.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/13decf0b-2b11-4031-8865-9650fb3fbda6_calibrate_994640_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:14 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:14 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:14 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:14 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:14 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 539 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb8e75f3d8c8bd4670e39f3ec6d73383 with result from task 539 parsl.dataflow.dflow DEBUG: Task 539 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/8acb7000-e2c2-406b-85dd-3c918c5c825a_calibrate_994638_22.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/8acb7000-e2c2-406b-85dd-3c918c5c825a_calibrate_994638_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:15 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:15 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:15 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:15 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:15 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80fe3a84e98a14a82aac80058c82886e with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/742d8152-1800-4d9a-ac9a-3f5c35a3e0dc_calibrate_994638_42.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/742d8152-1800-4d9a-ac9a-3f5c35a3e0dc_calibrate_994638_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:15 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:15 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06a85004a41ad6a28d24beec75840943 with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash d5e7ae122aca4bdc962cc3c8876f0dfe parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df66513a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 1081 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7fe1ca61-0b1d-4cb6-a404-eb1ed3c06d4b_calibrate_994639_7.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7fe1ca61-0b1d-4cb6-a404-eb1ed3c06d4b_calibrate_994639_7.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/9c29a172-125a-4be1-bfdb-3a92c5867729_characterizeImage_994639_7.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/9c29a172-125a-4be1-bfdb-3a92c5867729_characterizeImage_994639_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:15 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1085 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23a05d1a0eb1b4c54161e4f23d5687f7 with result from task 1085 parsl.dataflow.dflow DEBUG: Task 1085 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/615f5161-2e0a-46f5-b2ff-fff7c069529f_calibrate_994640_26.stdout parsl.dataflow.dflow INFO: Standard error for task 1085 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/615f5161-2e0a-46f5-b2ff-fff7c069529f_calibrate_994640_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:16 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:16 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:16 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98d4bbad57b5e039e734d9c21bfec797 with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/fc26323c-fe4b-458b-9e85-dd0cd80d01c8_calibrate_994638_34.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/fc26323c-fe4b-458b-9e85-dd0cd80d01c8_calibrate_994638_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:16 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:16 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 707c8ef7648d9efd443b39b4bebf63e7 with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/db919ba6-fe1b-4985-a633-214a47d536b3_calibrate_994638_47.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/db919ba6-fe1b-4985-a633-214a47d536b3_calibrate_994638_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:16 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49086b4ec4c4220632027a6146107e61 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/dc90d615-7a91-4fee-ac58-373e442d2fb9_calibrate_994639_36.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/dc90d615-7a91-4fee-ac58-373e442d2fb9_calibrate_994639_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:16 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 968 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69c114444c33486307f7fa1e5db1e810 with result from task 968 parsl.dataflow.dflow DEBUG: Task 968 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c424725c-cd7f-460b-9e08-2f5d5c287e48_calibrate_994639_43.stdout parsl.dataflow.dflow INFO: Standard error for task 968 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c424725c-cd7f-460b-9e08-2f5d5c287e48_calibrate_994639_43.stderr parsl.dataflow.dflow INFO: Task 1087 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcb8a177f1c95ffddbd2f50b8c90b4c9 with result from task 1087 parsl.dataflow.dflow DEBUG: Task 1087 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1088 has memoization hash b4e0750a4d368cfafc1c361b0d81faa5 parsl.dataflow.memoization INFO: Task 1088 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1088 try 0 launched on executor multi with executor id 1082 parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/91e9d6fe-c293-4452-8f5a-206e1a1181a7_calibrate_994638_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/91e9d6fe-c293-4452-8f5a-206e1a1181a7_calibrate_994638_58.stderr parsl.dataflow.dflow INFO: Standard output for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/10de8e7b-2a10-46f0-8872-0099a1cd51d2_characterizeImage_994638_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1087 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/10de8e7b-2a10-46f0-8872-0099a1cd51d2_characterizeImage_994638_58.stderr parsl.dataflow.dflow INFO: Task 1075 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a4e19f7f149df86c163c494bc52566d with result from task 1075 parsl.dataflow.dflow DEBUG: Task 1075 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1076 has memoization hash 21b4b2020119d5b0285732d377104a90 parsl.dataflow.memoization INFO: Task 1076 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ef420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1076 try 0 launched on executor multi with executor id 1083 parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/482f018f-5990-4abf-8f08-7a9f0c19c43f_calibrate_994638_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/482f018f-5990-4abf-8f08-7a9f0c19c43f_calibrate_994638_38.stderr parsl.dataflow.dflow INFO: Standard output for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/bffd2e48-52e4-404f-a803-736405ef1b5a_characterizeImage_994638_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1075 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/bffd2e48-52e4-404f-a803-736405ef1b5a_characterizeImage_994638_38.stderr parsl.dataflow.dflow INFO: Task 772 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d44cddf05022eda1600ad39636c350db with result from task 772 parsl.dataflow.dflow DEBUG: Task 772 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 773 has memoization hash 2e4236abd6eee0f779299b2ed143e499 parsl.dataflow.memoization INFO: Task 773 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 773 try 0 launched on executor multi with executor id 1084 parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/0bda8107-7f55-4f78-b0f1-1deca75b4f0f_calibrate_994638_12.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/0bda8107-7f55-4f78-b0f1-1deca75b4f0f_calibrate_994638_12.stderr parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/be4bd50e-c61f-4fb2-b9db-883bcd01b443_characterizeImage_994638_12.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/be4bd50e-c61f-4fb2-b9db-883bcd01b443_characterizeImage_994638_12.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4843c380ebc0897528a5f4c546c8212 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/04f6f649-71be-4aa2-a31f-c6dc110ce37a_calibrate_994640_20.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/04f6f649-71be-4aa2-a31f-c6dc110ce37a_calibrate_994640_20.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2d4cad2b3c7879346d9ffec5837aa15 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/102e28ee-a1ac-4614-8810-cf6bdc220049_calibrate_994638_60.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/102e28ee-a1ac-4614-8810-cf6bdc220049_calibrate_994638_60.stderr parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f13e34850c22282a215a9ebcdb62f3e with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 8c540c48c5ab0c261352e1e73df53158 parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6653ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 1085 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/fa3de252-59e3-44dd-b342-5b71a03b316c_calibrate_994640_54.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/fa3de252-59e3-44dd-b342-5b71a03b316c_calibrate_994640_54.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/20ebe835-f66e-41c4-8a6e-9c9f3fc5e836_characterizeImage_994640_54.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/20ebe835-f66e-41c4-8a6e-9c9f3fc5e836_characterizeImage_994640_54.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 810233a47ca08f84d049125018525fa0 with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f0b2c60c-742a-45e4-8fa2-1b605c37e17e_calibrate_994640_53.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f0b2c60c-742a-45e4-8fa2-1b605c37e17e_calibrate_994640_53.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46f46c98c438ddde2adbd9e4e09bd124 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash d5906b8d319e57cb183efa31d83c3998 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 1086 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/e9059dd8-d61a-469b-b08b-4942be4305c2_calibrate_994639_56.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/e9059dd8-d61a-469b-b08b-4942be4305c2_calibrate_994639_56.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/244f3313-a596-4bc4-bce8-b4d74cf727b2_characterizeImage_994639_56.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/244f3313-a596-4bc4-bce8-b4d74cf727b2_characterizeImage_994639_56.stderr parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 367d37b1d1a213a3d8782c1571835018 with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 08aa743425247be27de93e665f307648 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 1087 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/5d10d662-ca0d-4339-8073-782a22387a9c_calibrate_994639_17.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/5d10d662-ca0d-4339-8073-782a22387a9c_calibrate_994639_17.stderr parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/41d6c60d-0213-4037-b88f-648bd384721b_characterizeImage_994639_17.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/41d6c60d-0213-4037-b88f-648bd384721b_characterizeImage_994639_17.stderr parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65e727066978ecdfcd3ff71a85282df6 with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d881b651-c877-4b50-a3f1-9472a492c51c_calibrate_994638_55.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d881b651-c877-4b50-a3f1-9472a492c51c_calibrate_994638_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:19 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:19 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1001 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5e5f3c07959e0d944327d758f939c23 with result from task 1001 parsl.dataflow.dflow DEBUG: Task 1001 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2262daf0-4102-4900-a5db-b3ab1c790233_calibrate_994640_10.stdout parsl.dataflow.dflow INFO: Standard error for task 1001 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/2262daf0-4102-4900-a5db-b3ab1c790233_calibrate_994640_10.stderr parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38813d25f52f821ac8054b2a942dd487 with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash 76fca761adc5a11b6cdcdb57acdc5358 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64be0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 1088 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5668a3e3-3b0a-4024-b4f8-0ba67da91742_calibrate_994640_18.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5668a3e3-3b0a-4024-b4f8-0ba67da91742_calibrate_994640_18.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/1e87736f-7b05-4232-af58-757531e61f7e_characterizeImage_994640_18.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/1e87736f-7b05-4232-af58-757531e61f7e_characterizeImage_994640_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:19 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1093 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2baba51f2d5d99f623b2244630d50539 with result from task 1093 parsl.dataflow.dflow DEBUG: Task 1093 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1094 has memoization hash c71376f57e00af8b4808115ab0a6d076 parsl.dataflow.memoization INFO: Task 1094 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6344220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1094 try 0 launched on executor multi with executor id 1089 parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/14916898-aa25-41d1-b181-d3cb9e6b17b6_calibrate_994639_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/14916898-aa25-41d1-b181-d3cb9e6b17b6_calibrate_994639_49.stderr parsl.dataflow.dflow INFO: Standard output for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/54a2fbba-c726-4338-adaf-442383716b15_characterizeImage_994639_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1093 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/54a2fbba-c726-4338-adaf-442383716b15_characterizeImage_994639_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:19 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 958 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry daca98232a6dbe79952b9ed807725b76 with result from task 958 parsl.dataflow.dflow DEBUG: Task 958 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 959 has memoization hash e58022321e56318916c2d85c1e7caf3d parsl.dataflow.memoization INFO: Task 959 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df626b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 959 try 0 launched on executor multi with executor id 1090 parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cc50766d-9fde-4ea6-9b2e-fe051895d7ae_calibrate_994638_41.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cc50766d-9fde-4ea6-9b2e-fe051895d7ae_calibrate_994638_41.stderr parsl.dataflow.dflow INFO: Standard output for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9e0f7eb7-0a31-44c8-9247-bb7d17107610_characterizeImage_994638_41.stdout parsl.dataflow.dflow INFO: Standard error for task 958 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/9e0f7eb7-0a31-44c8-9247-bb7d17107610_characterizeImage_994638_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:20 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:20 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:20 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f07cee7d824749ecea3d0e3e67ae4508 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/89d81d5f-d166-4fe8-8729-c8e3c1696e67_calibrate_994640_43.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/89d81d5f-d166-4fe8-8729-c8e3c1696e67_calibrate_994640_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 62 parsl.dataflow.strategy DEBUG: Executor multi has 62 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:17:20 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:17:20 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:20 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:21 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:21 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:21 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:21 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:21 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:21 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:21 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:21 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:22 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:22 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:22 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dcfb1dc8326d9d7b3fab3f0ee3f0c10 with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/baf5ff65-22b9-4f53-a3bb-0ade4cf21f29_calibrate_994640_45.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/baf5ff65-22b9-4f53-a3bb-0ade4cf21f29_calibrate_994640_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:22 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:22 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:22 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:22 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:23 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:23 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f442f2f93a16460907e9f6c28e19d631 with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash e7fc307c27d438ee8b7b7878f3105bcd parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 1091 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/d0badd59-090d-4da3-b1fa-399d37532669_calibrate_994640_19.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/d0badd59-090d-4da3-b1fa-399d37532669_calibrate_994640_19.stderr parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/3b63b58e-c890-4a91-8625-5ecac3923d9d_characterizeImage_994640_19.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/3b63b58e-c890-4a91-8625-5ecac3923d9d_characterizeImage_994640_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:23 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:23 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:23 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:23 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:23 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:23 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:24 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:24 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:24 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 937 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbefec9a7a22f0894f580b02fc8dab66 with result from task 937 parsl.dataflow.dflow DEBUG: Task 937 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 938 has memoization hash efc0909cbace7bfa854834b149bc2a31 parsl.dataflow.memoization INFO: Task 938 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6269e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 938 try 0 launched on executor multi with executor id 1092 parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/a63ffaed-77ef-435f-9ab6-b49ffe3806c6_calibrate_994638_18.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/a63ffaed-77ef-435f-9ab6-b49ffe3806c6_calibrate_994638_18.stderr parsl.dataflow.dflow INFO: Standard output for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/26947abe-eb5f-4921-a26e-c09421121f23_characterizeImage_994638_18.stdout parsl.dataflow.dflow INFO: Standard error for task 937 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/26947abe-eb5f-4921-a26e-c09421121f23_characterizeImage_994638_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:24 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:25 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:25 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:25 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:25 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:25 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:25 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:25 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:25 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:26 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:26 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 1091 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c1767d84d3171f700beda3375b4a383 with result from task 1091 parsl.dataflow.dflow DEBUG: Task 1091 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/9fd165b9-321b-4fbb-b5ee-d42f5aef50ba_calibrate_994640_12.stdout parsl.dataflow.dflow INFO: Standard error for task 1091 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/9fd165b9-321b-4fbb-b5ee-d42f5aef50ba_calibrate_994640_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:26 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:26 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:26 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:26 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:26 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:27 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:27 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 586bcd792a80bde2a30689c56c3aaf32 with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/3b24fb30-ef49-43f2-bd14-265d8b8d36c2_calibrate_994639_48.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/3b24fb30-ef49-43f2-bd14-265d8b8d36c2_calibrate_994639_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:27 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:27 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:27 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:27 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:27 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:27 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:28 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:28 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:28 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:28 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:28 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:28 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 656fd9639960b0aaa883f22d80f57c1e with result from task 712 parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:28 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash 2ad1e68708de42ba7bc74e942103f983 parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 1093 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5a46374c-b54c-44dd-9cec-5cd3af71ae2f_calibrate_994640_29.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5a46374c-b54c-44dd-9cec-5cd3af71ae2f_calibrate_994640_29.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ea1eed1a-eae2-4107-965a-632e7affa3fa_characterizeImage_994640_29.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/ea1eed1a-eae2-4107-965a-632e7affa3fa_characterizeImage_994640_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:29 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:29 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2fd20f9d225e29e07d4777399ba2f22 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash ebb8779b46ce474675966a0941f88464 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6603920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 1094 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/3d22015a-9d20-4bdb-b46d-cc338f879873_calibrate_994638_53.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/3d22015a-9d20-4bdb-b46d-cc338f879873_calibrate_994638_53.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/d56796d4-a339-4428-94ef-507b069a3bbd_characterizeImage_994638_53.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/d56796d4-a339-4428-94ef-507b069a3bbd_characterizeImage_994638_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:29 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:29 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:29 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:29 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:29 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:30 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:30 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:30 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:30 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:30 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 1058 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d1ff7dcd577784cd7f4bd7f2e240201 with result from task 1058 parsl.dataflow.dflow DEBUG: Task 1058 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/f4ebd377-2325-4703-9ee7-b7abc6661f09_calibrate_994639_42.stdout parsl.dataflow.dflow INFO: Standard error for task 1058 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/f4ebd377-2325-4703-9ee7-b7abc6661f09_calibrate_994639_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:30 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:30 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:31 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:31 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:31 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65dfbd5ff6cc17f1201a0129ae4bff64 with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/2de62aea-9188-4b20-aa50-3b54f74abc56_calibrate_994638_57.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/2de62aea-9188-4b20-aa50-3b54f74abc56_calibrate_994638_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:31 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:31 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:31 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:31 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:31 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:32 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:32 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05979316784ad3ce65a26d8a3f89eb7f with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/3d1118d6-a38f-45a9-a592-18a7d9d586c4_calibrate_994639_60.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/3d1118d6-a38f-45a9-a592-18a7d9d586c4_calibrate_994639_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:32 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:32 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc0cdb09d3e835fa9703c0b535c23bd7 with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash c3982c80995a8c44954b7ad6881cf6ce parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63df740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 1095 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/32bdebe4-d9e0-488f-b81d-ed4406b65e3b_calibrate_994640_48.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/32bdebe4-d9e0-488f-b81d-ed4406b65e3b_calibrate_994640_48.stderr parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b993b482-107b-432e-94d5-b78e70dfdfc8_characterizeImage_994640_48.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/b993b482-107b-432e-94d5-b78e70dfdfc8_characterizeImage_994640_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:32 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:32 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:32 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:33 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:33 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e70d82f032581c1c49633667541f8b04 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c8330652-9fb2-4681-a0f1-180275601da1_calibrate_994639_47.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c8330652-9fb2-4681-a0f1-180275601da1_calibrate_994639_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:33 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:33 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 55 parsl.dataflow.strategy DEBUG: Executor multi has 55 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 55 parsl.dataflow.strategy DEBUG: Executor multi has 55 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17f89a395181cd4b83ec59cd8241a495 with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 2d06147e35e44025046406e213219fa5 parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df654e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 1096 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/3700ccd6-cba7-49c7-9727-808ee51b87f2_calibrate_994640_49.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/3700ccd6-cba7-49c7-9727-808ee51b87f2_calibrate_994640_49.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/1065cc2a-c864-4fb1-b3af-b8ec1aa6dd5b_characterizeImage_994640_49.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/1065cc2a-c864-4fb1-b3af-b8ec1aa6dd5b_characterizeImage_994640_49.stderr parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8e0dfe93fed216ed3b87f003b5a4d4e with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash 4a7fb8290385dc0385475e990e6fc63f parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 1097 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/8d7d45f8-d867-44f9-8261-1dcc25faaf60_calibrate_994640_55.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/8d7d45f8-d867-44f9-8261-1dcc25faaf60_calibrate_994640_55.stderr parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/66437cfd-f083-43f6-8540-4fba32ea09b6_characterizeImage_994640_55.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/66437cfd-f083-43f6-8540-4fba32ea09b6_characterizeImage_994640_55.stderr parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 158f3a7c5147eab64420853020ba9432 with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/0a86d611-efee-4f1a-b332-83a0b7459913_calibrate_994640_50.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/0a86d611-efee-4f1a-b332-83a0b7459913_calibrate_994640_50.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72d00df9c2c0b8ca3fb7315f23a47ea1 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5b07b8cc-af5b-45f9-a55a-8a2c5232f2c6_calibrate_994640_13.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5b07b8cc-af5b-45f9-a55a-8a2c5232f2c6_calibrate_994640_13.stderr parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fe7b6bf2417532be935b93af37f7f12 with result from task 803 parsl.dataflow.dflow DEBUG: Task 803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cc322e09-580f-4933-b020-59d11f6086de_calibrate_994638_49.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cc322e09-580f-4933-b020-59d11f6086de_calibrate_994638_49.stderr parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 831c86be99ca97107e1fd5453383198d with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ff1baab4-cf1d-49ae-8046-bcb4223b3565_calibrate_994638_39.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/ff1baab4-cf1d-49ae-8046-bcb4223b3565_calibrate_994638_39.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 51 parsl.dataflow.strategy DEBUG: Executor multi has 51 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 584161fc39e7d95640004e2b9c479fc8 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/1c1e538a-4709-4e5b-b173-d5e4ff99fe69_calibrate_994639_28.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/1c1e538a-4709-4e5b-b173-d5e4ff99fe69_calibrate_994639_28.stderr parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5708edac1d875c88dc77fdfcb6e7ed53 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash 556746ee8e87ab3006d76412ec1d9d7e parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6650fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 1098 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/57672f76-b87c-4000-b297-70260104a126_calibrate_994766_61.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/57672f76-b87c-4000-b297-70260104a126_calibrate_994766_61.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/afe2e65a-72d6-4e48-838d-4ce3ea97c3d6_characterizeImage_994766_61.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994766/afe2e65a-72d6-4e48-838d-4ce3ea97c3d6_characterizeImage_994766_61.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8f85394bb2d900f11fda7c174b1bed4 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 752dd3ea61cc91f6166ce7f625e9032c parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 1099 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/c2bb661a-ea2e-4dae-b645-2ebfbe0b8116_calibrate_994640_44.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/c2bb661a-ea2e-4dae-b645-2ebfbe0b8116_calibrate_994640_44.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/cb26b362-d62c-41c3-8eb9-1d62c76516e1_characterizeImage_994640_44.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/cb26b362-d62c-41c3-8eb9-1d62c76516e1_characterizeImage_994640_44.stderr parsl.dataflow.dflow INFO: Task 1078 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2af9e8306e22d11405d119a55beb3099 with result from task 1078 parsl.dataflow.dflow DEBUG: Task 1078 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1079 has memoization hash f64be546606f44654e991aa2911217bd parsl.dataflow.memoization INFO: Task 1079 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ef740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1079 try 0 launched on executor multi with executor id 1100 parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d9ae2494-153c-461e-bb8a-8c996f7378e2_calibrate_994638_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d9ae2494-153c-461e-bb8a-8c996f7378e2_calibrate_994638_62.stderr parsl.dataflow.dflow INFO: Standard output for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/8f268223-a8a8-4f6f-92d5-e5fc77164969_characterizeImage_994638_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1078 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/8f268223-a8a8-4f6f-92d5-e5fc77164969_characterizeImage_994638_62.stderr parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2b865f98fb9e203660b5a9d59ca2c53 with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8d0916f8-fb3a-48fd-9cad-e61b0d79d3bf_calibrate_994639_30.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/8d0916f8-fb3a-48fd-9cad-e61b0d79d3bf_calibrate_994639_30.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 49 parsl.dataflow.strategy DEBUG: Executor multi has 49 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de7bfaee788a820376693115a8107f3a with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/34b27ed5-0ede-4824-8609-e8b9aa620089_calibrate_994640_1.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/34b27ed5-0ede-4824-8609-e8b9aa620089_calibrate_994640_1.stderr parsl.dataflow.dflow INFO: Task 1052 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 481f9f7981ee1e1c861d9eaa276e2388 with result from task 1052 parsl.dataflow.dflow DEBUG: Task 1052 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/074aa2f0-a1f0-4f0d-866e-b569fca48158_calibrate_994639_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1052 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/074aa2f0-a1f0-4f0d-866e-b569fca48158_calibrate_994639_61.stderr parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1baf5e00dfeb50ec06b8ef9105e4064d with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 4933a4ed2111a4812446bc08b11a9d58 parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 1101 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/74b62ac8-80d1-47df-8b92-a0383b03dfb4_calibrate_994640_5.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/74b62ac8-80d1-47df-8b92-a0383b03dfb4_calibrate_994640_5.stderr parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/bfa2d793-4073-403b-b8f2-29e4408b3b24_characterizeImage_994640_5.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/bfa2d793-4073-403b-b8f2-29e4408b3b24_characterizeImage_994640_5.stderr parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0de569e57e416ef2fe760d7d646af4c0 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash 37bcdc59197caf964200a107ac003e36 parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df641d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 1102 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/f1ad99e2-e9e2-4c61-8e14-9f887301b9fe_calibrate_994638_23.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/f1ad99e2-e9e2-4c61-8e14-9f887301b9fe_calibrate_994638_23.stderr parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f7ba5002-edfc-48a5-b758-3efdb6d2d305_characterizeImage_994638_23.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/f7ba5002-edfc-48a5-b758-3efdb6d2d305_characterizeImage_994638_23.stderr parsl.dataflow.dflow INFO: Task 986 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b15e1f76a15a9e173fcc52d6e52e4d4a with result from task 986 parsl.dataflow.dflow DEBUG: Task 986 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/1836d583-e490-40b1-96b8-1f4c96908f4d_calibrate_994640_11.stdout parsl.dataflow.dflow INFO: Standard error for task 986 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/1836d583-e490-40b1-96b8-1f4c96908f4d_calibrate_994640_11.stderr parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32d5a0ebfaf0015d1a9d5c067f6b62f0 with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/04c905fa-e8ef-4a57-a4ee-16f15b711b8c_calibrate_994638_52.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/04c905fa-e8ef-4a57-a4ee-16f15b711b8c_calibrate_994638_52.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e45b17cae165986dea131b12cbb1e398 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c7d20f97-2e3e-468a-8fbc-dd09aad82566_calibrate_994639_32.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/c7d20f97-2e3e-468a-8fbc-dd09aad82566_calibrate_994639_32.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fcb18eabc190f34339f433c2a667678 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b20c70e0-f04f-4ad9-8501-ee543054fd49_calibrate_994640_14.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b20c70e0-f04f-4ad9-8501-ee543054fd49_calibrate_994640_14.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 43 parsl.dataflow.strategy DEBUG: Executor multi has 43 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0be005014c0386b0a5b6f6eafd5b1070 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash d01d3afaf1a80adc70a95b6f3aa06ee8 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df662c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 1103 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f19a2d97-d144-4c4b-8b5a-d10e9d63164b_calibrate_994640_23.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f19a2d97-d144-4c4b-8b5a-d10e9d63164b_calibrate_994640_23.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/a1d65427-2805-4157-852f-e270dd882009_characterizeImage_994640_23.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/a1d65427-2805-4157-852f-e270dd882009_characterizeImage_994640_23.stderr parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f23dd614621e24bb65b154e954dc50de with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4a8edad1-a023-4abb-b143-36acce90188f_calibrate_994640_57.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/4a8edad1-a023-4abb-b143-36acce90188f_calibrate_994640_57.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 151f2739f2d3bc4afd062c7c15f2b266 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d4cc9b26-b84d-47f6-a0fe-700cee37c926_calibrate_994638_8.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d4cc9b26-b84d-47f6-a0fe-700cee37c926_calibrate_994638_8.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d10b0cf4382757e73f67befa0ff0fa6 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/fa0cdc5c-3502-437a-84ee-fbba6af59680_calibrate_994639_34.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/fa0cdc5c-3502-437a-84ee-fbba6af59680_calibrate_994639_34.stderr parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b9a2004a715edc9b1a7a43110ed9fc6 with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/6300c284-eb8f-49c5-900b-c3d74a21c638_calibrate_994638_13.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/6300c284-eb8f-49c5-900b-c3d74a21c638_calibrate_994638_13.stderr parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acd9d325f8294d3ee8993f1c1b84768b with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/fe563bff-de37-4f03-a1af-69fe31687a4a_calibrate_994639_38.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/fe563bff-de37-4f03-a1af-69fe31687a4a_calibrate_994639_38.stderr parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c540c48c5ab0c261352e1e73df53158 with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/fa3de252-59e3-44dd-b342-5b71a03b316c_calibrate_994640_54.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/fa3de252-59e3-44dd-b342-5b71a03b316c_calibrate_994640_54.stderr parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4c42d15234ca5f2f5452ecf77e0d57c with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash 46786764d92e97fc34a6e116bfe8c160 parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 1104 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/09e2a301-cd3d-461e-9c12-e9e311826638_calibrate_994640_28.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/09e2a301-cd3d-461e-9c12-e9e311826638_calibrate_994640_28.stderr parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c16a81cb-c0e8-460d-90b2-c2fd6107ed3b_characterizeImage_994640_28.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/c16a81cb-c0e8-460d-90b2-c2fd6107ed3b_characterizeImage_994640_28.stderr 2024-03-15 04:17:53 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local parsl.dataflow.dflow INFO: Task 1076 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21b4b2020119d5b0285732d377104a90 with result from task 1076 parsl.dataflow.dflow DEBUG: Task 1076 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/482f018f-5990-4abf-8f08-7a9f0c19c43f_calibrate_994638_38.stdout parsl.dataflow.dflow INFO: Standard error for task 1076 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/482f018f-5990-4abf-8f08-7a9f0c19c43f_calibrate_994638_38.stderr 2024-03-15 04:17:53 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08aa743425247be27de93e665f307648 with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/5d10d662-ca0d-4339-8073-782a22387a9c_calibrate_994639_17.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/5d10d662-ca0d-4339-8073-782a22387a9c_calibrate_994639_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:54 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:54 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 959 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e58022321e56318916c2d85c1e7caf3d with result from task 959 parsl.dataflow.dflow DEBUG: Task 959 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cc50766d-9fde-4ea6-9b2e-fe051895d7ae_calibrate_994638_41.stdout parsl.dataflow.dflow INFO: Standard error for task 959 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/cc50766d-9fde-4ea6-9b2e-fe051895d7ae_calibrate_994638_41.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:55 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ff210479073c100b3a019c1eadbde58 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash cbe7b847439ede56233ffc7fd6b17205 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65e6de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 1105 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/4b8599fd-1c42-4e65-a367-90cbffb7d46f_calibrate_994638_29.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/4b8599fd-1c42-4e65-a367-90cbffb7d46f_calibrate_994638_29.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/afa2e822-1850-4ee3-bd38-b5b81e921ca2_characterizeImage_994638_29.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/afa2e822-1850-4ee3-bd38-b5b81e921ca2_characterizeImage_994638_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:55 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:55 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:56 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:56 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1094 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c71376f57e00af8b4808115ab0a6d076 with result from task 1094 parsl.dataflow.dflow DEBUG: Task 1094 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/14916898-aa25-41d1-b181-d3cb9e6b17b6_calibrate_994639_49.stdout parsl.dataflow.dflow INFO: Standard error for task 1094 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/14916898-aa25-41d1-b181-d3cb9e6b17b6_calibrate_994639_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:57 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:57 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 773 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e4236abd6eee0f779299b2ed143e499 with result from task 773 parsl.dataflow.dflow DEBUG: Task 773 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/0bda8107-7f55-4f78-b0f1-1deca75b4f0f_calibrate_994638_12.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/0bda8107-7f55-4f78-b0f1-1deca75b4f0f_calibrate_994638_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:58 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1066 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 603cd55301d06c642886429aca449444 with result from task 1066 parsl.dataflow.dflow DEBUG: Task 1066 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1067 has memoization hash 6b86e752f73d3b084032a3e6a2c5f726 parsl.dataflow.memoization INFO: Task 1067 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6345bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1067 try 0 launched on executor multi with executor id 1106 parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/9ef36ed6-acea-4a92-a2d9-f30d5f931a3c_calibrate_994638_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/9ef36ed6-acea-4a92-a2d9-f30d5f931a3c_calibrate_994638_37.stderr parsl.dataflow.dflow INFO: Standard output for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/d3225329-7277-481c-8fc7-b689d542403e_characterizeImage_994638_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1066 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/d3225329-7277-481c-8fc7-b689d542403e_characterizeImage_994638_37.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffdb35197f3edb2f80524a24468eaf05 with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/89e6d182-14ac-438d-9606-999cbf339a4c_calibrate_994638_5.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/89e6d182-14ac-438d-9606-999cbf339a4c_calibrate_994638_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:58 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:59 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:17:59 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:00 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7fc307c27d438ee8b7b7878f3105bcd with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/d0badd59-090d-4da3-b1fa-399d37532669_calibrate_994640_19.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/d0badd59-090d-4da3-b1fa-399d37532669_calibrate_994640_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:00 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:01 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1088 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4e0750a4d368cfafc1c361b0d81faa5 with result from task 1088 parsl.dataflow.dflow DEBUG: Task 1088 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/91e9d6fe-c293-4452-8f5a-206e1a1181a7_calibrate_994638_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1088 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/91e9d6fe-c293-4452-8f5a-206e1a1181a7_calibrate_994638_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:01 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:01 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5e7ae122aca4bdc962cc3c8876f0dfe with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7fe1ca61-0b1d-4cb6-a404-eb1ed3c06d4b_calibrate_994639_7.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/7fe1ca61-0b1d-4cb6-a404-eb1ed3c06d4b_calibrate_994639_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:02 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5906b8d319e57cb183efa31d83c3998 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/e9059dd8-d61a-469b-b08b-4942be4305c2_calibrate_994639_56.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/e9059dd8-d61a-469b-b08b-4942be4305c2_calibrate_994639_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:02 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 921a474913d852ea5ca2f0343ef9fbc0 with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash c9e6b1b4668beed665eef12cd9a2bb12 parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6393e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 1107 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5287fbcd-fee9-4d7e-a6e5-1e5d69bb568e_calibrate_994640_56.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5287fbcd-fee9-4d7e-a6e5-1e5d69bb568e_calibrate_994640_56.stderr parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/92a455c5-25dc-4289-9eb6-bc932840321f_characterizeImage_994640_56.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/92a455c5-25dc-4289-9eb6-bc932840321f_characterizeImage_994640_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:03 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:03 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:03 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:04 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebb8779b46ce474675966a0941f88464 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/3d22015a-9d20-4bdb-b46d-cc338f879873_calibrate_994638_53.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/3d22015a-9d20-4bdb-b46d-cc338f879873_calibrate_994638_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:04 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:04 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:04 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:04 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:05 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:05 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:05 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:05 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:05 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:05 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:05 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76fca761adc5a11b6cdcdb57acdc5358 with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5668a3e3-3b0a-4024-b4f8-0ba67da91742_calibrate_994640_18.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5668a3e3-3b0a-4024-b4f8-0ba67da91742_calibrate_994640_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:06 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:06 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:06 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:06 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:06 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:06 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:06 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:06 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:07 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:07 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:07 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:07 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:07 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:07 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:07 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:07 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:08 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:08 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:08 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:08 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:08 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:08 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:08 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 938 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efc0909cbace7bfa854834b149bc2a31 with result from task 938 parsl.dataflow.dflow DEBUG: Task 938 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/a63ffaed-77ef-435f-9ab6-b49ffe3806c6_calibrate_994638_18.stdout parsl.dataflow.dflow INFO: Standard error for task 938 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/a63ffaed-77ef-435f-9ab6-b49ffe3806c6_calibrate_994638_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:08 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:09 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:09 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:09 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:09 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:09 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:09 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:09 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:09 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:10 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:10 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:10 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:10 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:10 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:10 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:10 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ad1e68708de42ba7bc74e942103f983 with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5a46374c-b54c-44dd-9cec-5cd3af71ae2f_calibrate_994640_29.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5a46374c-b54c-44dd-9cec-5cd3af71ae2f_calibrate_994640_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:10 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:11 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:11 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:11 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:11 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:11 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:11 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:11 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:11 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:12 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea6ea571279d27cbd6e4a91141b7528b with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash ae4c19126ce691cda1e490cef12a0546 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df657eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 1108 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/f6e71ecf-2026-451d-94a5-ecff103177d0_calibrate_994639_44.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/f6e71ecf-2026-451d-94a5-ecff103177d0_calibrate_994639_44.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/48e24161-0321-4824-b601-90b6e4909e3d_characterizeImage_994639_44.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/48e24161-0321-4824-b601-90b6e4909e3d_characterizeImage_994639_44.stderr parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d06147e35e44025046406e213219fa5 with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/3700ccd6-cba7-49c7-9727-808ee51b87f2_calibrate_994640_49.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/3700ccd6-cba7-49c7-9727-808ee51b87f2_calibrate_994640_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:12 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:12 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:12 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:12 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:12 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:12 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:12 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 22 parsl.dataflow.strategy DEBUG: Executor multi has 22 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 22 parsl.dataflow.strategy DEBUG: Executor multi has 22 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 1115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51fc895f1af2578c81fe69bbaf5a2674 with result from task 1115 parsl.dataflow.dflow DEBUG: Task 1115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/acd388ff-8323-4d58-b286-354294dfcdef_calibrate_994640_47.stdout parsl.dataflow.dflow INFO: Standard error for task 1115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/acd388ff-8323-4d58-b286-354294dfcdef_calibrate_994640_47.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 752dd3ea61cc91f6166ce7f625e9032c with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/c2bb661a-ea2e-4dae-b645-2ebfbe0b8116_calibrate_994640_44.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/c2bb661a-ea2e-4dae-b645-2ebfbe0b8116_calibrate_994640_44.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 20 parsl.dataflow.strategy DEBUG: Executor multi has 20 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3982c80995a8c44954b7ad6881cf6ce with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/32bdebe4-d9e0-488f-b81d-ed4406b65e3b_calibrate_994640_48.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/32bdebe4-d9e0-488f-b81d-ed4406b65e3b_calibrate_994640_48.stderr parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4933a4ed2111a4812446bc08b11a9d58 with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/74b62ac8-80d1-47df-8b92-a0383b03dfb4_calibrate_994640_5.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/74b62ac8-80d1-47df-8b92-a0383b03dfb4_calibrate_994640_5.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 18 parsl.dataflow.strategy DEBUG: Executor multi has 18 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a328eb052cb498a03d539467c78dc6d with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash 88c769541afe249431eb6f60adf1bc79 parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6268f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 1109 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/98352c18-4942-409f-a09a-a21713673032_calibrate_994638_44.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/98352c18-4942-409f-a09a-a21713673032_calibrate_994638_44.stderr parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/739013a6-4dd3-4ec9-babc-e68a32441261_characterizeImage_994638_44.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/739013a6-4dd3-4ec9-babc-e68a32441261_characterizeImage_994638_44.stderr parsl.dataflow.dflow INFO: Task 1079 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f64be546606f44654e991aa2911217bd with result from task 1079 parsl.dataflow.dflow DEBUG: Task 1079 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d9ae2494-153c-461e-bb8a-8c996f7378e2_calibrate_994638_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1079 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/d9ae2494-153c-461e-bb8a-8c996f7378e2_calibrate_994638_62.stderr parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a7fb8290385dc0385475e990e6fc63f with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/8d7d45f8-d867-44f9-8261-1dcc25faaf60_calibrate_994640_55.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/8d7d45f8-d867-44f9-8261-1dcc25faaf60_calibrate_994640_55.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 556746ee8e87ab3006d76412ec1d9d7e with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/57672f76-b87c-4000-b297-70260104a126_calibrate_994766_61.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994766/57672f76-b87c-4000-b297-70260104a126_calibrate_994766_61.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91eea19ce8d980b87d19fe068737d37e with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash ac99e3a5a8c93e43d2e16c475207a2fa parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df64bdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 1110 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/16b73c65-db1d-448a-9cb8-5d211b0c9e97_calibrate_994638_40.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/16b73c65-db1d-448a-9cb8-5d211b0c9e97_calibrate_994638_40.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/39126770-b77d-4fb1-a9c6-3a07a8fd68a0_characterizeImage_994638_40.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/39126770-b77d-4fb1-a9c6-3a07a8fd68a0_characterizeImage_994638_40.stderr 2024-03-15 04:18:28 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:18:28 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:28 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:28 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:28 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:28 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:28 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:28 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:28 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:29 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:29 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:29 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:29 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:29 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:29 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:30 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:30 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:30 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:30 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1067 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b86e752f73d3b084032a3e6a2c5f726 with result from task 1067 parsl.dataflow.dflow DEBUG: Task 1067 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/9ef36ed6-acea-4a92-a2d9-f30d5f931a3c_calibrate_994638_37.stdout parsl.dataflow.dflow INFO: Standard error for task 1067 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/9ef36ed6-acea-4a92-a2d9-f30d5f931a3c_calibrate_994638_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:30 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cbe7b847439ede56233ffc7fd6b17205 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/4b8599fd-1c42-4e65-a367-90cbffb7d46f_calibrate_994638_29.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/4b8599fd-1c42-4e65-a367-90cbffb7d46f_calibrate_994638_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:30 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:30 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:31 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:31 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:31 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:31 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:31 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:31 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:31 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:32 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:32 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:32 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:32 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:32 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:32 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:32 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:32 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:33 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:33 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:33 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:33 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:33 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:33 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:33 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:34 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:34 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:34 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d01d3afaf1a80adc70a95b6f3aa06ee8 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f19a2d97-d144-4c4b-8b5a-d10e9d63164b_calibrate_994640_23.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/f19a2d97-d144-4c4b-8b5a-d10e9d63164b_calibrate_994640_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:34 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:34 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:34 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:34 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:34 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:35 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:35 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:35 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:35 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:35 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:35 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:35 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:36 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:36 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:36 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:36 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:36 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:36 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:36 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:37 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:37 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:37 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:37 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:37 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:37 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:37 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:38 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:38 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:38 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:38 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:38 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:38 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:38 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:38 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:39 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f0cd7f90d515eacde99243767912379 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash f95664a3450b91c59b64a6061abc6d97 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6602980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:39 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 1111 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9367521e-219e-41c6-9917-76421a150aac_calibrate_994639_62.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9367521e-219e-41c6-9917-76421a150aac_calibrate_994639_62.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/67a3f952-9dad-4ccc-b0a2-1f46e10b41b5_characterizeImage_994639_62.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994639/67a3f952-9dad-4ccc-b0a2-1f46e10b41b5_characterizeImage_994639_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:39 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:39 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:39 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:39 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:40 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:40 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:40 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:40 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:40 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:18:40 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37bcdc59197caf964200a107ac003e36 with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/f1ad99e2-e9e2-4c61-8e14-9f887301b9fe_calibrate_994638_23.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/f1ad99e2-e9e2-4c61-8e14-9f887301b9fe_calibrate_994638_23.stderr parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39ee442de2b9643f48aeaddc1d603edc with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash 6bb1bd3b376037908825bf475fca73b4 parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 1112 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/a353c098-4428-472e-a1b4-d26466de9642_calibrate_994638_11.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/a353c098-4428-472e-a1b4-d26466de9642_calibrate_994638_11.stderr parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/882e3ca2-460b-43bf-80c6-2393f27ffa69_characterizeImage_994638_11.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/882e3ca2-460b-43bf-80c6-2393f27ffa69_characterizeImage_994638_11.stderr parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46786764d92e97fc34a6e116bfe8c160 with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/09e2a301-cd3d-461e-9c12-e9e311826638_calibrate_994640_28.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/09e2a301-cd3d-461e-9c12-e9e311826638_calibrate_994640_28.stderr parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9e6b1b4668beed665eef12cd9a2bb12 with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5287fbcd-fee9-4d7e-a6e5-1e5d69bb568e_calibrate_994640_56.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/5287fbcd-fee9-4d7e-a6e5-1e5d69bb568e_calibrate_994640_56.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 9 parsl.dataflow.strategy DEBUG: Executor multi has 9 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae4c19126ce691cda1e490cef12a0546 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/f6e71ecf-2026-451d-94a5-ecff103177d0_calibrate_994639_44.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/f6e71ecf-2026-451d-94a5-ecff103177d0_calibrate_994639_44.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88c769541afe249431eb6f60adf1bc79 with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/98352c18-4942-409f-a09a-a21713673032_calibrate_994638_44.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/98352c18-4942-409f-a09a-a21713673032_calibrate_994638_44.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:19:00 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:19:00 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:01 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:01 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:01 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:01 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:01 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:01 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:01 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:02 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:02 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:02 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:02 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:02 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:02 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:02 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:02 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:03 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:03 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:03 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:03 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:03 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:03 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:03 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:03 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:04 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:04 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:04 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:04 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:04 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:04 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:04 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:05 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:05 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:05 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:05 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:05 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:05 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:05 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:05 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:06 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:06 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:06 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:06 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:06 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:06 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:06 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:07 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:07 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:07 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:07 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:07 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:07 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:07 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:08 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:08 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:08 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:08 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:08 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:08 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:08 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:08 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:09 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:09 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:09 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:09 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:09 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:09 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:09 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:09 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:10 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:10 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:10 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:10 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:10 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:10 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:10 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:11 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:11 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:11 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:11 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:11 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:11 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:11 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:12 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:12 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:12 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:12 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:12 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:12 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:12 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:12 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:13 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 7 parsl.dataflow.strategy DEBUG: Executor multi has 7 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac99e3a5a8c93e43d2e16c475207a2fa with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/16b73c65-db1d-448a-9cb8-5d211b0c9e97_calibrate_994638_40.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/16b73c65-db1d-448a-9cb8-5d211b0c9e97_calibrate_994638_40.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f95664a3450b91c59b64a6061abc6d97 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9367521e-219e-41c6-9917-76421a150aac_calibrate_994639_62.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994639/9367521e-219e-41c6-9917-76421a150aac_calibrate_994639_62.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bb1bd3b376037908825bf475fca73b4 with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/a353c098-4428-472e-a1b4-d26466de9642_calibrate_994638_11.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/a353c098-4428-472e-a1b4-d26466de9642_calibrate_994638_11.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0b2d412baa2319915cd95faa71cb998 with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash b3e630e30b9a70be55893ea456a6dd80 parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df63dd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 1113 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e7a1055c-7cdb-4d14-9746-c6f2d8f5bd6a_calibrate_994638_43.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e7a1055c-7cdb-4d14-9746-c6f2d8f5bd6a_calibrate_994638_43.stderr parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a0ee579b-7bde-407b-993f-9cb0cb3f084e_characterizeImage_994638_43.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/a0ee579b-7bde-407b-993f-9cb0cb3f084e_characterizeImage_994638_43.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:19:33 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:19:33 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:33 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:34 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:34 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:34 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:34 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:34 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:35 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:35 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:35 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:35 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:35 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:35 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:35 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:35 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:36 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:36 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:36 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:36 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:36 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:36 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:36 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:37 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:37 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:37 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:37 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:37 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:37 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:37 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:37 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:38 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:38 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:38 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:38 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:38 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:38 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:38 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:39 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:39 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:39 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:39 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:39 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:39 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:39 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:40 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:40 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:40 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:40 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:40 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:40 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:40 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:40 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:41 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:41 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:41 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:41 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:41 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:41 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:41 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:42 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:42 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:42 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:42 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:42 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:42 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:42 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:42 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:43 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:43 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:43 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:43 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:43 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:43 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:43 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:43 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:44 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:44 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:44 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:44 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:44 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:44 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:44 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:45 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:45 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:45 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:45 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:45 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:45 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:45 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:45 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:46 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:19:46 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3e630e30b9a70be55893ea456a6dd80 with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e7a1055c-7cdb-4d14-9746-c6f2d8f5bd6a_calibrate_994638_43.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/e7a1055c-7cdb-4d14-9746-c6f2d8f5bd6a_calibrate_994638_43.stderr 2024-03-15 04:20:06 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:20:06 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:07 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:07 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:07 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:07 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:08 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:08 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:08 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:08 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:08 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:08 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:08 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:09 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:09 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:09 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:09 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:09 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:09 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:09 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:09 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:10 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:10 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:10 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:10 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:10 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:10 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:10 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:11 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:11 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:11 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:11 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:11 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:11 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:11 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:11 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:12 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:12 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:12 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:12 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:12 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:12 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:12 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:13 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:13 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:13 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:13 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:13 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:13 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:13 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:13 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:14 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:14 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:14 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:14 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:14 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:14 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:14 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:14 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:15 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:15 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:15 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:15 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:15 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:15 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:15 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:16 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:16 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:16 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:16 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:16 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:16 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:16 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:17 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:17 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:17 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:17 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:17 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:17 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:17 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:17 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:18 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:18 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:18 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:18 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:18 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:18 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:18 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:19 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:19 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:19 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:19 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:19 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 229d9764ce103cdcf0bae98edbbc19df with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 72046e8fb0370878c4b0d55c0eecf810 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df65c3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 1114 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3be9e4c8-3da0-4c6b-8b8a-a12d1caa3099_calibrate_994767_61.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3be9e4c8-3da0-4c6b-8b8a-a12d1caa3099_calibrate_994767_61.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/a075fb17-8daf-4767-8818-085f36bca678_characterizeImage_994767_61.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994767/a075fb17-8daf-4767-8818-085f36bca678_characterizeImage_994767_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:20:39 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:20:40 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:41 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:41 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:41 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:41 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:41 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:41 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:42 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:42 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:42 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:42 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:42 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:42 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:42 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:42 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:43 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:43 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:43 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:43 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:43 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:43 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:43 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:43 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:44 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:44 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:44 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:44 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:44 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:44 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:44 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:45 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:45 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:45 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:45 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:45 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:45 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:45 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:45 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:46 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:46 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:46 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:46 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:46 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:46 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:46 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:46 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:47 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:47 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:47 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:47 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:47 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:47 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:47 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:48 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:48 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:48 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:48 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:48 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:48 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:48 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:48 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:49 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:49 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:49 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:49 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:49 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:49 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:49 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:50 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:50 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:50 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:50 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:50 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:50 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:50 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:50 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:51 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:51 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:51 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:51 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:51 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:51 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:51 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:51 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:52 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:52 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:52 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:52 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:52 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:52 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:52 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:20:53 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:21:13 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:21:13 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:13 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:13 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:13 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:14 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:14 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:14 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:14 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:14 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:15 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:15 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:15 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:15 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:15 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:15 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:15 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:16 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:16 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:16 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:16 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:16 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:16 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:16 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:16 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:17 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:17 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72046e8fb0370878c4b0d55c0eecf810 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3be9e4c8-3da0-4c6b-8b8a-a12d1caa3099_calibrate_994767_61.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994767/3be9e4c8-3da0-4c6b-8b8a-a12d1caa3099_calibrate_994767_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:17 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:17 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:17 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:17 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:17 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:18 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:18 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:18 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:18 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:18 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:18 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:18 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:19 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:19 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:19 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:19 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:19 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:19 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:19 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:20 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:20 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:20 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:20 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:20 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:20 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:20 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:21 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:21 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:21 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:21 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:21 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:21 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:21 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:22 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:22 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:22 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:22 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:22 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:22 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:22 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:22 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:23 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:23 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:23 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:23 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:23 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:23 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:23 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:24 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:24 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:24 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:24 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:24 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:24 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:24 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:24 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:25 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:25 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:25 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:25 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:25 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:25 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:25 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:25 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:26 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:26 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:21:46 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:21:46 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:46 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:46 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:46 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:47 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:47 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:48 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:48 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:48 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:48 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:48 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:48 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:48 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:48 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:49 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:49 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:49 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:49 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:49 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:49 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:49 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:50 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:50 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:50 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:50 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:50 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:50 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:50 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:51 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:51 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:51 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:51 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:51 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:51 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:51 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:51 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:52 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:52 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:52 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:52 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:52 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:52 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:52 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:52 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:53 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:53 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:53 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:53 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:53 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:53 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:53 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:53 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:54 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:54 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:54 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:54 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:54 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:54 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:54 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:55 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:55 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:55 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:55 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:55 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:55 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:55 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:55 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:56 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:56 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:56 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:56 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:56 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:56 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:56 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:57 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:57 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:57 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:57 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:57 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:57 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:57 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:58 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:58 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:58 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:58 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:58 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:58 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:58 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:58 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:59 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:59 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:21:59 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:22:19 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:22:19 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:19 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:19 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:20 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:20 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:21 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:21 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:21 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:21 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:22 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:22 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:22 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:22 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:22 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:22 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:22 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:23 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:23 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:23 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:23 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:23 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:23 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:23 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:23 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:24 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:24 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:24 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:24 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:24 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:24 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:24 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:25 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:25 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:25 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:25 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:25 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:25 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:26 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:26 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:26 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:26 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:26 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:26 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:26 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:26 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:27 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:27 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:27 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:27 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:27 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:27 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:27 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:28 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:28 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:28 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:28 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:28 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:28 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:28 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:28 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:29 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:29 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:29 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:29 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:29 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:29 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:30 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:30 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:30 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:30 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:30 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:30 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:30 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:30 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:31 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:31 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:31 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:31 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:31 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:31 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:31 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:32 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:32 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:32 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:32 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:32 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:32 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:32 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:32 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:33 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:33 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:33 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:22:53 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:22:53 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:53 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:53 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:53 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:54 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:55 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:55 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:55 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:55 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:55 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:55 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:56 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:56 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:56 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:56 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:56 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:56 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:56 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:56 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:57 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:57 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:57 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:57 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:57 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:57 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:57 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:58 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:58 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:58 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:58 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:58 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:58 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:58 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:58 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:59 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:59 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:59 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:59 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:59 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:59 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:59 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:22:59 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:00 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:00 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:00 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:00 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:00 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:00 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:00 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:01 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:01 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:01 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:01 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:01 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:01 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:01 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:01 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:02 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:02 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:02 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:02 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:02 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:02 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:03 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:03 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:03 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:03 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:03 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:03 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:03 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:03 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:04 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:04 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:04 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:04 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:04 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:04 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:04 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:04 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:05 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:05 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:05 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:05 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:05 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:05 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:05 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:06 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:06 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:06 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:06 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:06 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:06 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:23:26 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:23:26 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:27 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:27 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:27 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:28 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:29 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:29 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:29 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:29 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:29 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:30 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:30 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:30 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:30 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:30 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:30 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:30 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:30 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:31 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:31 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:31 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:31 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:31 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:31 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:31 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:32 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:32 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:32 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:32 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:32 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:32 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:32 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:32 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:33 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:33 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:33 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:33 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:33 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:33 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:33 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:34 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:34 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:34 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:34 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:34 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:34 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:34 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:34 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:35 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:35 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:35 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:35 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:35 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:35 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:35 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:36 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:36 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:36 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:36 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:36 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:36 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:36 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:36 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:37 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:37 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:37 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:37 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:37 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:37 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:37 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:38 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:38 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:38 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:38 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:38 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:38 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:38 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:39 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:39 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:39 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:39 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:39 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:39 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:39 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:39 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:40 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:40 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:40 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:40 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:40 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:40 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:23:41 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:24:01 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:24:01 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:01 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:01 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:01 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:03 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:03 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:03 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:03 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:03 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:03 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:04 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:04 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:04 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:04 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:04 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:04 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:04 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:05 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:05 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:05 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:05 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:05 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:05 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:05 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:06 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:06 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:06 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:06 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:06 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:06 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:07 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:07 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:07 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:07 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:07 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:07 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:07 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:08 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:08 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:08 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:08 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:08 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:08 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:08 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:09 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:09 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:09 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:09 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:09 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:09 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:09 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:09 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:10 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:10 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:10 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:10 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:10 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:10 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:10 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:11 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:11 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:11 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:11 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:11 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:11 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:11 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:11 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:12 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:12 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:12 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:12 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:12 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:12 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:12 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:13 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:13 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:13 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:13 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:13 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:13 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:13 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:13 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:14 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:14 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:14 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:14 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:14 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:14 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:14 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:15 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:15 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:15 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:24:35 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:24:35 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:35 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:37 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:37 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:37 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:37 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:37 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:38 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:38 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:38 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:38 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:38 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:38 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:38 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:39 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:39 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:39 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:39 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:39 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:39 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:39 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:39 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:40 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:40 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:40 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:40 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:40 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:40 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:40 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:40 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:41 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:41 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:41 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:41 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:41 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:41 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:41 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:41 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:42 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:42 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:42 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:42 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:42 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:42 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:42 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:43 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:43 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:43 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:43 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:43 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:43 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:43 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:43 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:44 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:44 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:44 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:44 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:44 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:44 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:44 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:45 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:45 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:45 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:45 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:45 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:45 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:45 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:45 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:46 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:46 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:46 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:46 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:46 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:46 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:46 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:47 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:47 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:47 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:47 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:47 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:47 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:47 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:47 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:48 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:48 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:48 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:48 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:48 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:48 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:48 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:49 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:24:49 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:25:09 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:25:09 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:09 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:09 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:09 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:11 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:11 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:11 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:11 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:11 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:12 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:12 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:12 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:12 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:12 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:12 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:12 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:13 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:13 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:13 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:13 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:13 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:13 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:13 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:13 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:14 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:14 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:14 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:14 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:14 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:14 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:15 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:15 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:15 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:15 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:15 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:15 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:15 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:16 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:16 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:16 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:16 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:16 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:16 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:16 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:16 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:17 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:17 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:17 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:17 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:17 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:17 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:17 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:18 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:18 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:18 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:18 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:18 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:18 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:18 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:18 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:19 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:19 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:19 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:19 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:19 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:19 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:19 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:20 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:20 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:20 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:20 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:20 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:20 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:20 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:20 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:21 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:21 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:21 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:21 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:21 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:21 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:21 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:22 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:22 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:22 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:22 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:22 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:22 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:22 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:22 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:23 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:23 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:25:43 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:25:43 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:43 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:43 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:43 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:43 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:44 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:44 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:44 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:44 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:45 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:45 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:45 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:45 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:45 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:45 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:46 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:46 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:46 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:46 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:46 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:46 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:46 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:46 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:47 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:47 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:47 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:47 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:47 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:47 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:47 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:48 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:48 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:48 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:48 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:48 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:48 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:48 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:49 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:49 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:49 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:49 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:49 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:49 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:49 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:49 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:50 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:50 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:50 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:50 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:50 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:50 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:50 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:50 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:51 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:51 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:51 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:51 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:51 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:51 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:51 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:52 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:52 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:52 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:52 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:52 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:52 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:52 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:52 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:53 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:53 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:53 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:53 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:53 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:53 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:53 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:53 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:54 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:54 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:54 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:54 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:54 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:54 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:54 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:55 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:55 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:55 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:55 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:55 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:55 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:55 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:55 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:25:56 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:26:16 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:26:16 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:16 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:16 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:16 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:16 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:17 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:17 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:18 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:18 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:18 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:18 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:18 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:18 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:18 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:19 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:19 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:19 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:19 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:19 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:19 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:19 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:19 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:20 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:20 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:20 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:20 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:20 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:20 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:20 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:21 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:21 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:21 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:21 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:21 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:21 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:21 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:21 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:22 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:22 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:22 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:22 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:22 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:22 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:22 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:22 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:23 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:23 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:23 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:23 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:23 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:23 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:24 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:24 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:24 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:24 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:24 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:24 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:24 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:24 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:25 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:25 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:25 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:25 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:25 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:25 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:25 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:26 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:26 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:26 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:26 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:26 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:26 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:27 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:27 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:27 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:27 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:27 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:27 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:27 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:27 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:28 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:28 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:28 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:28 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:28 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:28 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:29 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:29 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:29 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:29 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:29 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:30 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:26:50 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:26:50 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:51 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:51 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:51 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:51 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:52 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:52 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:52 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:52 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:52 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:52 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:52 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:52 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:53 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:53 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:53 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:53 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:53 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:53 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:53 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:54 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:54 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:54 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:54 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:54 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:54 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:54 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:55 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:55 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:55 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:55 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:55 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:55 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:55 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:55 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:56 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:56 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:56 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:56 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:56 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:56 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:56 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:57 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:57 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:57 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:57 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:57 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:57 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:57 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:57 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:58 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:58 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:58 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:58 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:58 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:58 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:58 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:58 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:59 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:59 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:59 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:59 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:59 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:59 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:26:59 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:00 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:00 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:00 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:00 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:00 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:00 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:00 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:00 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:01 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:01 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:01 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:01 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:01 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:01 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:01 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:02 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:02 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:02 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:02 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:02 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:02 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:02 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:02 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:03 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:03 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:03 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:03 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:27:23 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:27:23 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:23 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:24 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:24 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:24 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:24 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:24 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:25 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:25 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:25 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:25 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:25 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:25 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:25 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:25 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:26 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:26 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:26 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:26 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:26 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:26 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:26 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:27 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:27 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:27 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:27 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:27 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:27 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:27 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:27 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:28 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:28 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:28 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:28 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:28 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:28 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:28 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:28 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:29 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:29 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:29 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:29 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:30 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:30 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:30 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:30 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:30 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:30 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:30 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:31 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:31 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:31 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:31 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:31 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:31 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:31 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:32 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:32 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:32 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:32 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:32 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:32 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:32 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:32 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:33 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:33 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:33 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:33 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:33 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:33 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:33 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:34 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:34 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:34 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:34 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:34 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:34 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:34 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:34 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:35 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:35 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:35 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:35 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:35 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:35 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:35 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:36 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:36 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:36 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:36 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:36 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:36 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:27:56 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:27:56 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:57 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:57 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:57 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:57 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:57 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:58 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:58 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:58 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:58 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:58 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:58 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:58 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:58 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:59 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:59 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:59 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:59 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:59 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:59 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:59 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:27:59 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:00 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:00 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:00 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:00 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:00 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:00 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:00 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:01 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:01 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:01 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:01 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:01 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:01 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:01 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:01 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:02 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:02 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:02 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:02 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:02 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:02 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:02 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:02 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:03 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:03 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:03 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:03 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:03 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:03 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:03 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:04 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:04 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:04 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:04 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:04 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:04 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:04 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:05 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:05 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:05 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:05 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:05 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:05 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:05 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:06 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:06 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:06 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:06 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:06 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:06 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:06 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:07 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:07 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:07 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:07 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:07 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:07 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:07 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:08 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:08 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:08 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:08 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:08 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:08 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:08 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:08 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:09 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:09 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:09 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:09 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:28:29 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:28:29 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:29 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:30 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:30 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:30 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:30 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:30 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:31 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:31 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:31 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:31 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:31 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:31 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:31 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:32 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:32 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:32 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:32 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:32 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:32 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:32 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:32 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:33 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:33 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:33 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:33 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:33 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:33 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:33 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:34 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:34 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:34 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:34 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:34 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:34 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:34 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:34 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:35 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:35 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:35 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:35 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:35 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:35 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:35 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:36 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:36 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:36 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:36 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:36 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:36 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:36 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:36 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:37 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:37 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:37 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:37 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:37 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:37 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:37 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:38 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:38 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:38 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:38 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:38 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:38 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:38 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:38 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:39 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:39 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:39 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:39 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:39 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:39 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:39 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:40 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:40 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:40 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:40 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:40 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:40 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:40 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:40 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:41 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:41 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:41 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:41 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:41 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:41 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:41 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:42 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:42 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:28:42 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:29:02 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:29:02 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:02 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:02 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:02 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:03 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:03 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:04 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:04 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:04 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:04 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:04 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:04 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:04 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:04 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:05 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:05 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:05 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:05 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:05 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:05 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:05 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:06 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:06 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:06 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:06 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:06 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:06 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:06 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:06 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:07 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:07 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:07 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:07 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:07 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:07 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:07 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:07 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:08 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:08 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:08 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:08 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:08 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:08 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:08 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:09 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:09 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:09 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:09 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:09 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:09 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:09 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:09 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:10 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:10 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:10 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:10 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:10 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:10 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:10 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:11 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:11 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:11 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:11 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:11 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:11 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:11 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:11 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:12 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:12 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:12 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:12 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:12 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:12 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:12 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:13 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:13 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:13 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:13 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:13 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:13 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:13 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:13 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:14 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:14 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:14 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:14 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:14 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:14 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:14 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:15 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:15 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:15 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:29:35 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:29:35 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:35 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:35 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:36 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:36 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:37 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:37 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:37 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:37 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:37 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:37 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:37 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:38 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:38 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:38 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:38 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:38 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:38 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:38 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:39 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:39 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:39 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:39 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:39 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:39 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:39 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:39 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:40 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:40 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:40 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:40 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:40 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:40 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:40 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:40 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:41 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:41 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:41 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:41 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:41 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:41 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:42 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:42 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:42 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:42 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:42 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:42 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:42 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:42 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:43 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:43 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:43 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:43 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:43 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:43 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:43 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:44 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:44 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:44 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:44 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:44 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:44 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:44 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:44 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:45 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:45 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:45 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:45 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:45 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:45 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:45 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:46 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:46 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:46 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:46 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:46 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:46 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:46 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:46 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:47 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:47 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:47 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:47 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:47 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:47 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:47 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:47 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:48 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:48 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:48 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:48 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:29:48 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:30:08 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:30:08 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:09 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:09 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:09 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:09 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:09 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:10 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:10 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:10 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:10 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:10 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:10 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:10 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:11 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:11 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:11 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:11 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:11 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:11 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:11 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:11 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:12 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:12 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:12 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:12 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:12 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:12 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:12 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:12 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:13 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:13 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:13 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:13 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:13 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:13 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:13 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:14 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:14 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:14 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:14 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:14 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:14 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:14 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:14 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:15 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:15 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:15 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:15 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:15 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:15 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:15 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:15 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:16 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:16 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:16 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:16 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:16 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:16 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:16 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:17 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:17 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:17 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:17 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:17 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:17 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:17 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:17 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:18 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:18 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:18 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:18 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:18 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:18 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:18 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:19 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:19 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:19 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:19 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:19 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:19 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:19 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:19 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:20 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:20 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:20 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:20 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:20 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:20 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:20 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:21 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:21 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:21 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:30:41 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:30:41 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:41 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:42 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:42 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:42 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:43 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:43 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:43 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:44 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:44 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:44 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:44 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:44 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:44 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:44 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:44 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:45 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:45 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:45 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:45 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:45 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:45 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:45 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:46 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:46 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:46 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:46 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:46 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:46 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:46 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:46 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:47 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:47 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:47 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:47 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:47 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:47 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:47 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:47 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:48 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:48 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:48 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:48 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:48 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:48 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:48 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:49 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:49 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:49 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:49 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:49 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:49 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:49 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:49 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:50 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:50 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:50 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:50 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:50 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:50 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:50 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:50 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:51 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:51 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:51 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:51 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:51 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:51 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:51 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:52 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:52 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:52 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:52 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:52 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:52 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:52 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:53 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:53 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:53 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:53 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:53 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:53 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:53 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:53 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:54 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:54 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:54 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:54 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:54 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:54 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:54 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:30:54 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:31:15 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:31:15 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:15 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:15 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:15 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:16 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:17 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:17 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:17 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:17 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:18 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:18 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:18 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:18 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:18 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:18 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:18 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:19 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:19 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:19 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:19 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:19 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:19 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:19 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:19 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:20 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:20 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:20 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:20 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:20 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:20 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:20 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:21 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:21 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:21 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:21 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:21 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:21 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:21 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:21 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:22 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:22 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:22 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:22 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:22 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:22 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:22 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:22 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:23 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:23 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:23 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:23 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:23 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:23 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:23 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:24 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:24 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:24 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:24 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:24 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:24 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:24 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:24 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:25 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:25 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:25 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:25 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:25 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:25 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:25 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:26 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:26 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:26 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:26 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:26 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:26 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:26 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:26 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:27 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:27 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:27 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:27 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:27 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:27 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:27 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:28 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:28 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:28 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:28 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:28 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:28 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:28 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:28 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:31:49 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:31:49 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:49 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:49 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:49 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:49 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:50 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:51 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:51 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:51 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:51 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:51 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:51 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:51 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:52 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:52 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:52 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:52 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:52 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:52 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:52 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:52 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:53 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:53 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:53 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:53 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:53 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:53 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:53 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:53 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:54 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:54 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:54 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:54 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:54 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:54 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:54 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:55 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:55 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:55 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:55 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:55 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:55 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:55 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:55 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:56 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:56 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:56 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:56 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:56 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:56 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:56 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:56 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:57 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:57 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:57 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:57 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:57 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:57 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:57 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:58 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:58 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:58 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:58 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:58 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:58 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:58 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:58 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:59 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:59 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:59 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:59 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:59 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:59 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:59 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:31:59 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:00 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:00 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:00 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:00 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:00 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:00 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:00 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:01 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:01 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:01 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:01 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:01 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:01 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:01 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:01 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:02 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:02 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:32:22 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:32:22 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:23 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:23 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:23 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:24 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:24 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:24 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:24 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:24 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:25 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:25 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:25 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:25 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:25 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:25 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:25 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:25 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:26 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:26 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:26 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:26 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:26 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:26 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:26 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:27 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:27 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:27 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:27 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:27 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:27 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:27 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:27 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:28 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:28 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:28 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:28 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:28 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:28 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:28 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:28 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:29 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:29 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:29 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:29 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:29 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:29 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:30 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:30 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:30 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:30 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:30 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:30 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:31 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:31 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:31 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:31 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:31 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:31 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:32 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:32 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:32 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:32 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:32 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:32 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:32 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:33 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:33 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:33 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:33 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:33 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:33 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:34 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:34 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:34 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:34 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:34 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:34 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:34 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:34 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:35 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:35 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:35 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:35 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:35 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:35 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:35 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:36 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:36 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:36 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:36 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:36 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:36 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:32:56 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:32:56 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:57 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:58 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:58 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:58 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:58 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:58 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:58 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:58 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:59 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:59 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:59 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:59 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:59 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:59 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:59 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:32:59 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:00 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:00 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:00 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:00 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:00 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:00 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:00 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:00 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:01 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:01 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:01 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:01 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:01 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:01 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:01 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:02 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:02 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:02 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:02 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:02 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:02 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:02 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:02 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:03 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:03 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:03 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:03 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:03 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:03 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:03 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:03 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:04 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:04 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:04 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:04 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:04 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:04 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:04 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:05 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:05 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:05 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:05 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:05 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:05 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:05 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:05 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:06 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:06 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:06 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:06 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:06 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:06 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:06 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:06 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:07 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:07 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:07 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:07 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:07 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:07 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:07 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:08 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:08 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:08 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:08 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:08 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:08 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:08 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:09 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:09 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:09 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:09 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:09 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:09 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:09 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:33:29 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:33:30 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:30 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:31 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:31 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:31 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:31 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:31 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:32 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:32 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:32 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:32 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:32 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:32 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:32 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:33 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:33 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:33 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:33 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:33 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:33 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:33 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:33 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:34 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:34 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:34 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:34 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:34 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:34 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:34 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:35 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:35 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:35 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:35 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:35 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:35 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:35 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:35 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:36 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:36 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:36 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:36 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:36 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:36 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:36 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:37 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:37 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:37 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:37 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:37 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:37 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:37 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:37 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:38 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:38 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:38 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:38 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:38 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:38 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:38 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:39 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:39 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:39 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:39 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:39 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:39 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:39 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:39 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:40 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:40 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:40 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:40 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:40 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:40 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:40 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:41 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:41 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:41 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:41 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:41 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:41 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:41 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:41 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:42 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:42 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:42 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:42 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:42 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:42 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:42 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:42 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:43 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:33:43 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:34:03 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:34:03 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:03 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:03 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:03 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:05 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:05 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:05 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:05 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:05 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:05 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:06 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:06 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:06 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:06 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:06 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:06 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:06 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:07 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:07 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:07 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:07 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:07 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:07 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:07 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:07 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:08 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:08 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:08 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:08 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:08 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:08 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:08 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:08 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:09 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:09 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:09 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:09 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:09 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:09 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:09 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:10 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:10 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:10 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:10 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:10 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:10 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:10 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:10 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:11 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:11 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:11 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:11 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:11 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:11 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:11 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:11 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:12 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:12 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:12 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:12 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:12 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:12 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:12 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:13 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:13 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:13 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:13 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:13 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:13 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:13 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:13 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:14 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:14 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:14 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:14 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:14 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:14 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:14 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:15 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:15 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:15 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:15 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:15 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:15 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:15 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:15 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:16 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:16 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:16 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:16 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:16 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:16 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:34:36 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:34:36 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:37 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:37 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:37 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:37 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:38 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:38 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:38 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:38 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:38 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:38 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:38 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:38 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:39 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:39 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:39 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:39 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:39 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:39 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:39 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:40 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:40 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:40 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:40 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:40 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:40 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:40 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:40 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:41 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:41 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:41 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:41 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:41 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:41 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:41 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:41 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:42 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:42 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:42 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:42 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:42 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:42 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:42 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:43 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:43 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:43 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:43 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:43 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:43 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:43 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:43 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:44 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:44 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:44 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:44 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:44 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:44 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:44 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:44 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:45 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:45 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:45 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:45 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:45 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:45 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:45 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:46 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:46 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:46 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:46 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:46 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:46 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:46 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:46 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:47 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:47 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:47 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:47 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:47 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:47 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:47 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:48 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:48 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:48 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:48 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:48 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:48 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:48 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:48 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:49 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:49 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:34:49 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:35:09 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:35:09 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:09 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:10 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:10 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:10 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:10 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:11 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:11 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:11 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:11 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:11 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:11 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:11 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:12 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:12 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:12 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:12 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:12 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:12 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:12 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:12 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:13 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:13 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:13 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:13 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:13 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:13 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:13 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:14 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:14 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:14 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:14 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:14 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:14 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:14 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:14 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:15 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:15 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:15 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:15 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:15 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:15 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:15 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:16 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:16 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:16 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:16 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:16 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:16 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:16 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:16 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:17 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:17 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:17 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:17 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:17 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:17 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:17 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:18 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:18 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:18 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:18 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:18 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:18 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:18 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:19 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:19 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:19 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:19 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:19 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:19 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:19 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:20 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:20 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:20 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:20 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:20 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:20 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:20 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:21 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:21 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:21 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:21 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:21 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:21 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:21 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:22 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:22 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:22 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:22 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:22 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:22 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:35:43 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:35:43 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:43 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:43 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:43 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:43 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:43 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:43 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:44 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:44 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:44 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:44 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:44 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:44 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:44 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:44 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:45 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:45 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:45 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:45 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:45 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:46 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:46 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:46 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:46 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:46 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:46 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:46 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:46 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:47 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:47 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:47 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:47 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:47 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:47 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:47 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:47 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:48 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:48 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:48 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:48 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:48 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:48 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:48 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:49 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:49 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:49 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:49 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:49 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:49 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:50 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:50 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:50 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:50 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:50 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:50 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:50 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:50 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:51 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:51 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:51 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:51 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:51 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:51 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:51 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:52 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:52 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:52 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:52 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:52 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:52 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:52 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:52 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:53 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:53 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:53 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:53 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:53 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:53 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:53 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:54 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:54 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:54 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:54 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:54 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:54 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:54 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:54 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:55 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:55 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:55 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:55 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:35:55 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:36:15 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:36:15 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:16 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:16 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:16 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:16 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:17 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:17 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:17 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:17 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:17 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:17 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:17 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:18 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:18 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:18 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:18 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:18 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:18 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:18 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:18 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:19 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:19 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:19 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:19 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:19 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:19 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:19 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:20 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:20 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:20 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:20 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:20 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:20 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:20 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:20 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:21 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:21 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:21 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:21 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:21 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:21 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:21 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:21 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:22 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:22 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:22 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:22 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:22 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:22 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:22 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:23 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:23 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:23 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:23 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:23 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:23 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:23 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:23 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:24 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:24 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:24 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:24 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:24 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:24 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:24 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:25 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:25 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:25 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:25 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:25 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:25 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:25 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:25 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:26 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:26 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:26 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:26 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:26 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:26 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:26 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:27 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:27 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:27 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:27 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:27 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:27 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:27 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:27 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:28 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:28 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:28 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:28 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:36:48 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:36:48 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:48 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:49 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:50 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:50 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:50 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:50 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:50 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:50 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:50 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:50 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:51 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:51 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:51 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:51 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:51 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:51 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:51 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:51 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:52 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:52 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:52 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:52 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:52 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:52 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:52 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:53 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:53 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:53 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:53 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:53 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:53 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:53 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:54 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:54 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:54 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:54 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:54 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:54 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:54 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:55 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:55 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:55 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:55 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:55 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:55 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:55 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:55 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:56 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:56 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:56 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:56 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:56 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:56 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:56 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:57 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:57 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:57 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:57 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:57 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:57 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:57 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:57 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:58 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:58 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:58 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:58 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:58 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:58 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:58 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:59 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:59 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:59 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:59 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:59 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:59 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:59 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:36:59 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:00 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:00 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:00 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:00 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:00 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:00 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:01 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:01 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:01 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:01 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:01 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:01 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:01 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:01 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:37:22 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:37:22 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:22 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:23 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:23 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:23 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:23 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:23 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:24 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:24 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:24 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:24 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:24 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:24 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:24 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:25 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:25 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:25 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:25 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:25 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:25 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:25 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:25 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:26 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:26 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:26 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:26 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:26 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:26 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:26 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:27 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:27 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:27 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:27 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:27 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:27 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:27 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:27 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:28 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:28 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:28 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:28 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:28 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:28 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:28 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:28 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:29 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:29 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:29 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:29 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:29 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:30 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:30 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:30 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:30 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:30 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:30 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:30 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:30 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:31 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:31 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:31 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:31 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:31 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:31 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:32 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:32 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:32 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:32 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:32 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:32 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:32 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:32 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:33 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:33 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:33 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:33 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:33 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:33 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:33 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:34 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:34 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:34 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:34 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:34 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:34 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:34 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:34 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:35 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:35 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:35 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:35 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:35 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:37:55 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:37:55 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:57 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:57 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:57 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:57 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:57 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:57 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:57 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:58 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:58 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:58 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:58 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:58 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:58 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:58 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:59 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:59 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:59 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:59 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:59 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:59 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:59 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:37:59 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:00 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:00 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:00 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:00 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:00 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:00 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:00 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:01 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:01 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:01 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:01 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:01 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:01 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:01 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:01 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:02 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:02 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:02 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:02 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:02 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:02 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:02 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:02 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:03 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:03 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:03 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:03 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:03 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:03 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:03 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:04 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:04 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:04 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:04 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:04 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:04 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:04 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:04 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:05 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:05 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:05 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:05 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:05 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:05 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:05 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:06 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:06 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:06 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:06 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:06 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:06 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:06 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:06 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:07 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:07 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:07 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:07 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:07 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:07 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:07 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:08 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:08 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:08 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:08 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:08 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:08 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:08 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:08 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:09 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:38:29 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:38:29 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:29 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:29 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:30 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:31 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:31 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:31 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:31 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:31 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:31 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:31 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:32 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:32 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:32 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:32 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:32 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:32 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:32 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:33 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:33 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:33 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:33 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:33 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:33 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:33 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:33 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:34 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:34 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:34 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:34 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:34 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:34 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:34 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:34 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:35 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:35 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:35 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:35 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:35 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:35 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:35 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:36 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:36 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:36 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:36 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:36 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:36 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:36 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:36 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:37 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:37 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:37 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:37 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:37 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:37 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:37 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:38 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:38 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:38 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:38 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:38 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:38 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:38 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:38 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:39 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:39 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:39 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:39 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:39 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:39 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:39 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:40 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:40 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:40 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:40 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:40 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:40 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:40 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:40 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:41 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:41 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:41 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:41 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:41 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:41 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:41 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:42 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:42 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:42 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:42 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:42 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:38:42 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:39:02 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:39:02 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:03 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:03 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:03 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:03 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:04 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:04 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:05 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:05 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:05 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:05 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:05 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:05 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:05 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:06 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:06 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:06 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:06 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:06 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:06 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:06 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:06 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:07 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:07 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:07 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:07 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:07 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:07 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:07 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:07 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:08 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:08 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:08 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:08 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:08 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:08 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:08 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:09 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:09 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:09 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:09 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:09 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:09 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:09 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:09 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:10 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:10 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:10 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:10 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:10 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:10 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:10 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:10 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:11 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:11 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:11 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:11 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:11 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:11 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:11 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:12 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:12 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:12 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:12 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:12 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:12 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:12 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:12 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:13 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:13 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:13 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:13 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:13 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:13 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:13 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:14 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:14 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:14 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:14 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:14 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:14 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:14 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:14 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:15 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:15 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:15 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:15 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:15 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:15 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:15 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:16 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:16 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:39:36 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:39:36 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:36 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:36 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:37 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:38 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:38 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:38 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:38 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:39 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:39 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:39 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:39 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:39 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:39 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:39 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:39 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:40 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:40 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:40 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:40 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:40 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:40 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:40 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:41 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:41 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:41 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:41 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:41 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:41 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:41 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:41 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:42 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:42 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:42 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:42 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:42 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:42 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:43 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:43 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:43 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:43 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:43 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:43 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:43 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:43 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:44 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:44 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:44 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:44 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:44 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:44 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:45 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:45 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:45 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:45 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:45 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:45 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:45 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:46 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:46 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:46 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:46 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:46 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:46 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:46 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:46 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:47 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:47 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:47 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:47 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:47 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:47 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:47 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:47 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:48 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:48 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:48 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:48 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:48 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:48 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:48 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:49 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:49 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:49 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:49 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:49 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:49 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:49 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:50 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:50 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:50 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:39:50 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 1054 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d472406b7f3f5b6831404850fc0b697 with result from task 1054 parsl.dataflow.dflow DEBUG: Task 1054 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1055 has memoization hash e5b2ea0ba2ceaac80685b5a569709bef parsl.dataflow.memoization INFO: Task 1055 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df62ee2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1055 try 0 launched on executor multi with executor id 1115 parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c33ea51a-fae2-4e51-b409-d7a1d4c85518_calibrate_994638_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c33ea51a-fae2-4e51-b409-d7a1d4c85518_calibrate_994638_61.stderr parsl.dataflow.dflow INFO: Standard output for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/dcf33530-7751-46d9-b4ae-a1d4e0c84738_characterizeImage_994638_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1054 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994638/dcf33530-7751-46d9-b4ae-a1d4e0c84738_characterizeImage_994638_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:40:10 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:40:10 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:10 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:10 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:12 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:12 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:12 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:12 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:12 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:12 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:13 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:13 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:13 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:13 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:13 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:13 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:13 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:13 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:14 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:14 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:14 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:14 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:14 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:14 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:14 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:15 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:15 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:15 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:15 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:15 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:15 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:15 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:15 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:16 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:16 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:16 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:16 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:16 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:16 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:16 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:17 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:17 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:17 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:17 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:17 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:17 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:17 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:18 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:18 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:18 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:18 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:18 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:18 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:18 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:18 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:19 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:19 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:19 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:19 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:19 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:19 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:19 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:20 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:20 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:20 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:20 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:20 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:20 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:20 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:20 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:21 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:21 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:21 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:21 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:21 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:21 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:21 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:22 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:22 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:22 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:22 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:22 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:22 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:22 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:22 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:23 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:23 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:23 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:23 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:23 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:23 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:23 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:23 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:40:44 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:40:44 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:44 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:44 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:44 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:44 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:44 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:45 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.dataflow.dflow INFO: Task 1055 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5b2ea0ba2ceaac80685b5a569709bef with result from task 1055 parsl.dataflow.dflow DEBUG: Task 1055 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c33ea51a-fae2-4e51-b409-d7a1d4c85518_calibrate_994638_61.stdout parsl.dataflow.dflow INFO: Standard error for task 1055 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994638/c33ea51a-fae2-4e51-b409-d7a1d4c85518_calibrate_994638_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:45 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:45 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:45 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:45 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:45 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:45 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:46 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:46 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:46 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:46 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:46 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:46 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:47 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:47 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:47 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:47 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:47 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:47 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:47 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:47 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:48 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:48 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:48 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:48 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:48 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:48 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:48 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:49 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:49 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:49 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:49 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:49 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:49 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:49 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:49 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:50 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:50 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:50 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:50 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:50 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:50 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:50 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:50 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:51 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:51 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:51 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:51 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:51 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:51 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:51 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:52 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:52 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:52 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:52 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:52 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:52 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:53 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:53 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:53 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:53 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:53 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:53 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:53 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:54 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:54 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:54 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:54 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:54 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:54 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:54 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:54 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:55 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:55 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:55 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:55 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:55 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:55 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:55 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:56 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:56 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:56 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:56 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:56 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:56 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:40:56 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:41:16 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:41:17 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:17 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:17 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:17 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:18 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:18 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:18 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:18 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:18 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:18 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:19 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:19 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:19 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:19 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:19 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:19 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:19 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:19 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:20 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:20 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:20 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:20 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:20 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:20 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:20 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:20 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:21 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:21 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:21 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:21 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:21 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:21 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:21 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:22 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:22 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:22 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:22 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:22 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:22 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:22 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:22 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:23 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:23 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:23 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:23 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:23 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:23 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:23 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:24 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:24 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:24 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:24 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:24 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:24 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:24 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:24 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:25 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:25 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:25 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:25 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:25 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:25 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:25 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:26 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:26 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:26 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:26 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:26 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:26 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:26 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:26 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:27 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:27 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:27 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:27 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:27 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:27 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:27 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:28 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:28 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:28 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:28 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:28 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:28 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:29 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:29 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:29 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:29 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:29 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:29 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:30 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:30 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:41:50 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:41:50 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:50 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:50 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:51 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:52 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:52 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:52 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:52 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:52 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:52 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:52 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:53 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:53 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:53 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:53 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:53 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:53 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:53 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:53 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:54 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:54 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:54 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:54 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:54 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:54 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:55 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:55 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:55 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:55 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:55 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:55 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:55 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:56 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:56 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:56 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:56 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:56 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:56 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:56 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:56 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:57 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:57 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:57 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:57 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:57 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:57 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:57 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:58 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:58 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:58 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:58 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:58 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:58 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:58 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:58 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:59 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:59 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:59 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:59 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:59 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:59 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:59 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:41:59 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:00 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:00 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:00 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:00 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:00 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:00 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:00 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:01 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:01 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:01 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:01 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:01 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:01 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:01 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:01 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:02 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:02 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:02 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:02 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:02 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:02 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:02 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:02 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:03 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:03 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:03 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:03 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:03 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:03 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:42:23 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:42:23 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:24 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:24 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:24 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:24 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:24 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:24 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:25 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:25 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:25 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:25 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:25 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:25 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:25 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:25 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:26 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:26 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:26 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:26 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:26 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:26 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:26 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:27 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:27 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:27 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:27 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:27 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:27 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:27 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:27 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:28 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:28 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:28 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:28 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:28 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:28 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:28 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:28 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:29 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:29 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:29 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:29 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:29 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:29 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:29 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:30 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:30 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:30 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:30 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:30 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:30 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:31 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:31 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:31 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:31 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:31 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:31 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:31 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:32 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:32 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:32 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:32 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:32 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:32 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:32 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:32 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:33 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:33 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:33 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:33 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:33 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:33 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:33 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:34 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:34 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:34 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:34 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:34 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:34 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:34 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:34 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:35 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:35 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:35 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:35 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:35 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:35 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:35 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:36 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:36 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:36 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:36 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:42:56 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:42:56 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:56 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:56 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:57 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:57 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:57 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:57 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:57 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:57 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:57 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:58 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:58 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:58 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:58 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:58 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:58 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:58 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:58 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:59 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:59 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:59 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:59 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:59 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:59 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:59 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:42:59 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:00 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:00 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:00 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:00 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:00 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:00 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:00 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:01 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:01 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:01 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:01 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:01 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:01 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:01 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:01 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:02 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:02 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:02 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:02 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:02 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:02 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:02 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:02 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:03 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:03 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:03 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:03 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:03 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:03 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:03 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:04 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:04 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:04 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:04 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:04 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:04 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:04 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:04 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:05 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:05 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:05 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:05 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:05 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:05 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:05 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:06 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:06 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:06 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:06 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:06 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:06 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:06 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:06 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:07 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:07 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:07 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:07 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:07 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:07 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:07 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:08 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:08 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:08 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:08 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:08 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:08 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:43:29 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:43:29 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:29 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:29 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:30 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:30 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:30 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:30 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:30 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:30 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:31 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:31 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:31 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:31 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:31 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:31 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:31 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:32 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:32 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:32 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:32 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:32 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:32 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:32 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:32 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:33 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:33 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:33 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:33 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:33 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:33 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:33 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:33 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:34 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:34 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:34 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:34 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:34 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:34 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:34 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:35 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:35 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:35 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:35 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:35 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:35 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:35 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:36 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:36 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:36 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:36 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:36 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:36 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:36 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:36 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:37 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:37 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:37 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:37 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:37 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:37 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:37 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:38 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:38 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:38 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:38 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:38 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:38 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:38 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:38 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:39 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:39 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:39 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:39 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:39 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:39 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:39 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:40 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:40 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:40 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:40 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:40 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:40 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:40 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:40 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:41 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:41 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:41 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:41 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:41 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:41 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:41 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:43:42 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27aaa1272bcb2d74ae25e1b58e932544 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash abe3bc8cec25a609929197fd8faa4d0a parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153df6484680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 1116 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b235b039-687d-4439-be4d-b8cfb422aef7_calibrate_994640_61.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b235b039-687d-4439-be4d-b8cfb422aef7_calibrate_994640_61.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2766688a-e143-4ac2-af32-b84b52a76b29_characterizeImage_994640_61.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/characterizeImage/994640/2766688a-e143-4ac2-af32-b84b52a76b29_characterizeImage_994640_61.stderr 2024-03-15 04:44:02 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:44:02 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:02 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:02 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:02 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:02 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:03 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:03 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:03 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:03 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:03 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:03 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:03 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:03 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:04 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:04 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:04 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:04 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:04 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:04 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:04 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:05 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:05 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:05 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:05 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:05 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:05 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:05 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:05 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:06 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:06 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:06 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:06 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:06 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:06 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:06 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:07 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:07 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:07 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:07 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:07 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:07 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:07 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:08 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:08 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:08 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:08 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:08 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:08 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:08 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:09 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:09 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:09 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:09 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:09 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:09 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:09 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:09 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:10 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:10 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:10 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:10 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:10 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:10 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:10 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:10 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:11 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:11 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:11 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:11 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:11 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:11 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:11 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:12 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:12 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:12 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:12 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:12 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:12 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:12 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:12 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:13 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:13 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:13 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:13 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:13 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:13 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:13 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:13 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:14 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:14 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:14 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:14 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 2024-03-15 04:44:34 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abe3bc8cec25a609929197fd8faa4d0a with result from task 488 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 1104 parsl.dataflow.dflow INFO: Tasks in state States.failed: 6 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 6 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b235b039-687d-4439-be4d-b8cfb422aef7_calibrate_994640_61.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/submit/DEEP/20210507/A0j/science#step1/20240315T041214Z/logs/calibrate/994640/b235b039-687d-4439-be4d-b8cfb422aef7_calibrate_994640_61.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller 2024-03-15 04:44:34 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:34 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:34 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:35 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:35 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:35 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:35 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:36 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:36 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:36 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:36 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:36 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:36 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:36 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:36 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:37 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:37 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:37 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:37 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:37 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:37 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:38 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:38 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:38 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:38 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:38 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:38 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:38 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:38 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:39 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057287,17057288,17057289,17057290,17057291,17057292,17057293,17057294,17057295,17057296,17057297,17057298,17057299,17057300,17057301,17057302,17057303,17057304,17057305,17057306,17057307,17057308,17057309,17057310,17057311,17057312,17057313,17057314,17057315,17057316' parsl.providers.slurm.slurm DEBUG: squeue returned 17057309 R 17057310 R 17057311 R 17057312 R 17057313 R 17057314 R 17057315 R 17057316 R 17057299 R 17057300 R 17057301 R 17057302 R 17057303 R 17057304 R 17057305 R 17057306 R 17057307 R 17057308 R 17057289 R 17057290 R 17057291 R 17057292 R 17057293 R 17057294 R 17057295 R 17057296 R 17057297 R 17057298 R 17057287 R 17057288 R parsl.providers.slurm.slurm DEBUG: Updating job 17057309 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057310 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057311 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057312 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057313 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057314 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057315 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057316 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057299 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057300 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057301 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057302 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057303 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057304 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057305 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057306 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057307 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057308 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057289 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057290 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057291 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057292 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057293 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057294 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057295 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057296 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057297 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057298 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057287 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057288 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:39 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:39 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:39 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:39 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:39 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057317,17057318,17057319,17057320,17057321' parsl.providers.slurm.slurm DEBUG: squeue returned 17057317 R 17057318 R 17057319 R 17057320 R 17057321 R parsl.providers.slurm.slurm DEBUG: Updating job 17057317 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057318 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057319 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057320 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057321 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:39 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:40 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:40 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:40 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:40 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:40 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:40 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:40 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:40 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:41 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:41 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:41 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:41 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:41 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:41 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:41 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:42 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:42 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:42 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:42 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057322,17057323,17057324,17057325,17057326,17057327,17057328,17057329,17057330,17057331,17057332,17057333,17057334,17057335,17057336,17057337,17057338,17057339,17057340,17057341' parsl.providers.slurm.slurm DEBUG: squeue returned 17057335 R 17057336 R 17057337 R 17057338 R 17057339 R 17057340 R 17057341 R 17057325 R 17057326 R 17057327 R 17057328 R 17057329 R 17057330 R 17057331 R 17057332 R 17057333 R 17057334 R 17057322 R 17057323 R 17057324 R parsl.providers.slurm.slurm DEBUG: Updating job 17057335 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057336 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057337 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057338 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057339 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057340 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057341 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057325 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057326 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057327 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057328 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057329 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057330 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057331 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057332 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057333 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057334 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057322 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057323 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057324 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:42 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:42 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:42 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:42 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:43 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:43 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:43 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:43 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:43 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:43 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:43 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:44 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:44 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:44 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:44 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:44 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:44 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:44 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:44 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:45 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:45 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:45 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:45 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:45 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:45 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:45 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:45 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:46 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:46 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:46 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:46 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:46 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:46 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:46 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:47 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:47 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING 2024-03-15 04:44:47 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17057342,17057343,17057344,17057345,17057346,17057347,17057348,17057349,17057350,17057351,17057352,17057353,17057354,17057355,17057356,17057357,17057358,17057359,17057360,17057361,17057362,17057363,17057364,17057365,17057366,17057367,17057368,17057369,17057370,17057371,17057372,17057373,17057374,17057375,17057376,17057377,17057378' parsl.providers.slurm.slurm DEBUG: squeue returned 17057342 R 17057343 R 17057344 R 17057345 R 17057346 R 17057347 R 17057348 R 17057349 R 17057350 R 17057351 R 17057352 R 17057353 R 17057354 R 17057355 R 17057356 R 17057357 R 17057358 R 17057359 R 17057360 R 17057361 R 17057362 R 17057363 R 17057364 R 17057365 R 17057366 R 17057367 R 17057368 R 17057369 R 17057370 R 17057371 R 17057372 R 17057373 R 17057374 R 17057375 R 17057376 R 17057377 R 17057378 R parsl.providers.slurm.slurm DEBUG: Updating job 17057342 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057343 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057344 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057345 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057346 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057347 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057348 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057349 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057350 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057351 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057352 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057353 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057354 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057355 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057356 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057357 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057358 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057359 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057360 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057361 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057362 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057363 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057364 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057365 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057366 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057367 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057368 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057369 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057370 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057371 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057372 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057373 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057374 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057375 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057376 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057377 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17057378 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 93, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 93/0 running/pending blocks, and 353 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (93) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 3.814697265625e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23355963767952 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=93, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 93 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c50166b9f3a2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c50166b9f3a2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5aaa34afe8c1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5aaa34afe8c1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: da4498aae174 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: da4498aae174 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1701bb50967e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1701bb50967e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 76842a08adc8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 76842a08adc8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7c49ddb5c499 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7c49ddb5c499 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a1a7bc1bfd33 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a1a7bc1bfd33 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f61979c98a09 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f61979c98a09 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 12699c5fe935 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 12699c5fe935 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 843cf8a36284 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 843cf8a36284 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7be0ab4d5f23 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7be0ab4d5f23 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 610a18ee8623 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 610a18ee8623 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 770055b99597 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 770055b99597 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 990a0d711ee0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 990a0d711ee0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a3d275c11a94 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a3d275c11a94 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a0e7a9e520a2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a0e7a9e520a2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: df6b317efe93 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: df6b317efe93 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d43fa5875bd5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d43fa5875bd5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 329256d060b8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 329256d060b8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f15144a17adc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f15144a17adc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0322e9065e69 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0322e9065e69 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 824e8cd48200 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 824e8cd48200 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 566b4b333d6b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 566b4b333d6b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3e541890bdac parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3e541890bdac parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b7553d2ea111 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b7553d2ea111 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 17b37126b601 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 17b37126b601 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9e09ac5a3573 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9e09ac5a3573 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1b908727b103 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1b908727b103 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f730d2da1e64 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f730d2da1e64 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d554b94a5428 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d554b94a5428 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 683021a445bf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 683021a445bf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9449202abbf4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9449202abbf4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: def5ca522632 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: def5ca522632 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0f304c809809 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0f304c809809 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e76315eb7ef5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e76315eb7ef5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4b59f700d46f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4b59f700d46f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 151079f76be5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 151079f76be5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d79458dbede6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d79458dbede6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a866a29aea00 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a866a29aea00 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 456582003199 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 456582003199 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c8613eda9e0f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c8613eda9e0f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 434f2990e6e1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 434f2990e6e1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2de5955a1c88 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2de5955a1c88 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5983a981f917 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5983a981f917 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 50dbdb0be68b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 50dbdb0be68b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d12e8d71ddb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d12e8d71ddb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fd7daca5b81b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fd7daca5b81b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 14096c6dece7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 14096c6dece7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3d6ec95a1bd3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3d6ec95a1bd3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f834c63b999e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f834c63b999e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3ace99012ef6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3ace99012ef6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 665f5ad2eaf5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 665f5ad2eaf5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 420f731eed3b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 420f731eed3b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c1fc28d80d3d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c1fc28d80d3d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0d4a6d7b7f46 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0d4a6d7b7f46 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4a7e082ce472 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4a7e082ce472 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f74357ef5fa5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f74357ef5fa5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 108a7af2ab6e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 108a7af2ab6e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3d9e83b0a211 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3d9e83b0a211 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a3fccbb73d27 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a3fccbb73d27 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5bc217b21e74 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5bc217b21e74 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e29e10c85922 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e29e10c85922 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a6cd86705f5c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a6cd86705f5c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f8bfa7f2e325 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f8bfa7f2e325 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 24b1c450597e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 24b1c450597e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 712ab6959f8e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 712ab6959f8e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c34c7c097aff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c34c7c097aff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2cc1ee8ba2c6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2cc1ee8ba2c6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3193d5e0a1f9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3193d5e0a1f9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e9d4846f66fa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e9d4846f66fa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 637688c69ff5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 637688c69ff5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dc5b4f0da878 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dc5b4f0da878 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f68e15a16186 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f68e15a16186 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 95adc6d210e6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 95adc6d210e6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: aa684c845b20 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: aa684c845b20 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: be0461202248 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: be0461202248 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dfb39b4b3d41 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dfb39b4b3d41 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b2bc9161c4d4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b2bc9161c4d4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f4fc3a46649d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f4fc3a46649d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 09a47fb1f769 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 09a47fb1f769 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 621029ee9efc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 621029ee9efc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9a6e2f4c9bf9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9a6e2f4c9bf9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5b9b23ab7463 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5b9b23ab7463 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e03391b5c04c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e03391b5c04c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3006f5049d70 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3006f5049d70 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ae50dae4f9c8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ae50dae4f9c8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7548c8d6ca83 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7548c8d6ca83 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e4713cf15df7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e4713cf15df7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7b01352179d9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7b01352179d9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8f7402712a42 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8f7402712a42 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 809137215ba2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 809137215ba2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a88042aa7c45 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a88042aa7c45 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 05ce6db4cef2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 05ce6db4cef2 2024-03-15 04:44:47 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint 2024-03-15 04:44:47 proc_lsst.multi:201 [INFO] cancelling 17057344 on provider checkpoint proc_lsst.multi INFO: cancelling 17057344 on provider checkpoint 2024-03-15 04:44:47 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint 2024-03-15 04:44:47 proc_lsst.multi:201 [INFO] cancelling 17057362 on provider checkpoint proc_lsst.multi INFO: cancelling 17057362 on provider checkpoint 2024-03-15 04:44:47 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint 2024-03-15 04:44:47 proc_lsst.multi:201 [INFO] cancelling 17057347 on provider checkpoint proc_lsst.multi INFO: cancelling 17057347 on provider checkpoint 2024-03-15 04:44:47 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint 2024-03-15 04:44:47 proc_lsst.multi:201 [INFO] cancelling 17057366 on provider checkpoint proc_lsst.multi INFO: cancelling 17057366 on provider checkpoint 2024-03-15 04:44:47 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 2024-03-15 04:44:47 proc_lsst.multi:201 [INFO] cancelling 17057339 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057339 on provider escience_a40 2024-03-15 04:44:48 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 2024-03-15 04:44:48 proc_lsst.multi:201 [INFO] cancelling 17057332 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057332 on provider escience_a40 2024-03-15 04:44:48 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro 2024-03-15 04:44:48 proc_lsst.multi:201 [INFO] cancelling 17057310 on provider astro proc_lsst.multi INFO: cancelling 17057310 on provider astro 2024-03-15 04:44:48 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint 2024-03-15 04:44:48 proc_lsst.multi:201 [INFO] cancelling 17057355 on provider checkpoint proc_lsst.multi INFO: cancelling 17057355 on provider checkpoint 2024-03-15 04:44:48 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint 2024-03-15 04:44:48 proc_lsst.multi:201 [INFO] cancelling 17057359 on provider checkpoint proc_lsst.multi INFO: cancelling 17057359 on provider checkpoint 2024-03-15 04:44:48 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 2024-03-15 04:44:48 proc_lsst.multi:201 [INFO] cancelling 17057324 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057324 on provider escience_a40 2024-03-15 04:44:48 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint 2024-03-15 04:44:48 proc_lsst.multi:201 [INFO] cancelling 17057364 on provider checkpoint proc_lsst.multi INFO: cancelling 17057364 on provider checkpoint 2024-03-15 04:44:48 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint 2024-03-15 04:44:48 proc_lsst.multi:201 [INFO] cancelling 17057370 on provider checkpoint proc_lsst.multi INFO: cancelling 17057370 on provider checkpoint 2024-03-15 04:44:48 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 2024-03-15 04:44:48 proc_lsst.multi:201 [INFO] cancelling 17057322 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057322 on provider escience_a40 2024-03-15 04:44:48 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro 2024-03-15 04:44:48 proc_lsst.multi:201 [INFO] cancelling 17057294 on provider astro proc_lsst.multi INFO: cancelling 17057294 on provider astro 2024-03-15 04:44:48 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint 2024-03-15 04:44:48 proc_lsst.multi:201 [INFO] cancelling 17057361 on provider checkpoint proc_lsst.multi INFO: cancelling 17057361 on provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057321 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17057321 on provider escience_rtx6k 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057356 on provider checkpoint proc_lsst.multi INFO: cancelling 17057356 on provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057371 on provider checkpoint proc_lsst.multi INFO: cancelling 17057371 on provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057343 on provider checkpoint proc_lsst.multi INFO: cancelling 17057343 on provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057378 on provider checkpoint proc_lsst.multi INFO: cancelling 17057378 on provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057288 on provider astro proc_lsst.multi INFO: cancelling 17057288 on provider astro 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 55323 on provider local proc_lsst.multi INFO: cancelling 55323 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 55323 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057352 on provider checkpoint proc_lsst.multi INFO: cancelling 17057352 on provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057340 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057340 on provider escience_a40 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057357 on provider checkpoint proc_lsst.multi INFO: cancelling 17057357 on provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057323 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057323 on provider escience_a40 2024-03-15 04:44:49 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint 2024-03-15 04:44:49 proc_lsst.multi:201 [INFO] cancelling 17057373 on provider checkpoint proc_lsst.multi INFO: cancelling 17057373 on provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:201 [INFO] cancelling 17057348 on provider checkpoint proc_lsst.multi INFO: cancelling 17057348 on provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:201 [INFO] cancelling 17057354 on provider checkpoint proc_lsst.multi INFO: cancelling 17057354 on provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro 2024-03-15 04:44:50 proc_lsst.multi:201 [INFO] cancelling 17057290 on provider astro proc_lsst.multi INFO: cancelling 17057290 on provider astro 2024-03-15 04:44:50 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro 2024-03-15 04:44:50 proc_lsst.multi:201 [INFO] cancelling 17057303 on provider astro proc_lsst.multi INFO: cancelling 17057303 on provider astro 2024-03-15 04:44:50 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro 2024-03-15 04:44:50 proc_lsst.multi:201 [INFO] cancelling 17057312 on provider astro proc_lsst.multi INFO: cancelling 17057312 on provider astro 2024-03-15 04:44:50 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:201 [INFO] cancelling 17057365 on provider checkpoint proc_lsst.multi INFO: cancelling 17057365 on provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:201 [INFO] cancelling 17057351 on provider checkpoint proc_lsst.multi INFO: cancelling 17057351 on provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 2024-03-15 04:44:50 proc_lsst.multi:201 [INFO] cancelling 17057335 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057335 on provider escience_a40 2024-03-15 04:44:50 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:201 [INFO] cancelling 17057342 on provider checkpoint proc_lsst.multi INFO: cancelling 17057342 on provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint 2024-03-15 04:44:50 proc_lsst.multi:201 [INFO] cancelling 17057376 on provider checkpoint proc_lsst.multi INFO: cancelling 17057376 on provider checkpoint 2024-03-15 04:44:51 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro 2024-03-15 04:44:51 proc_lsst.multi:201 [INFO] cancelling 17057304 on provider astro proc_lsst.multi INFO: cancelling 17057304 on provider astro 2024-03-15 04:44:51 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 2024-03-15 04:44:51 proc_lsst.multi:201 [INFO] cancelling 17057337 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057337 on provider escience_a40 2024-03-15 04:44:51 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro 2024-03-15 04:44:51 proc_lsst.multi:201 [INFO] cancelling 17057306 on provider astro proc_lsst.multi INFO: cancelling 17057306 on provider astro 2024-03-15 04:44:51 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k 2024-03-15 04:44:51 proc_lsst.multi:201 [INFO] cancelling 17057317 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17057317 on provider escience_rtx6k 2024-03-15 04:44:51 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro 2024-03-15 04:44:51 proc_lsst.multi:201 [INFO] cancelling 17057301 on provider astro proc_lsst.multi INFO: cancelling 17057301 on provider astro 2024-03-15 04:44:51 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint 2024-03-15 04:44:51 proc_lsst.multi:201 [INFO] cancelling 17057360 on provider checkpoint proc_lsst.multi INFO: cancelling 17057360 on provider checkpoint 2024-03-15 04:44:51 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 2024-03-15 04:44:51 proc_lsst.multi:201 [INFO] cancelling 17057341 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057341 on provider escience_a40 2024-03-15 04:44:51 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint 2024-03-15 04:44:51 proc_lsst.multi:201 [INFO] cancelling 17057358 on provider checkpoint proc_lsst.multi INFO: cancelling 17057358 on provider checkpoint 2024-03-15 04:44:52 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro 2024-03-15 04:44:52 proc_lsst.multi:201 [INFO] cancelling 17057289 on provider astro proc_lsst.multi INFO: cancelling 17057289 on provider astro 2024-03-15 04:44:52 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint 2024-03-15 04:44:52 proc_lsst.multi:201 [INFO] cancelling 17057363 on provider checkpoint proc_lsst.multi INFO: cancelling 17057363 on provider checkpoint 2024-03-15 04:44:52 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro 2024-03-15 04:44:52 proc_lsst.multi:201 [INFO] cancelling 17057295 on provider astro proc_lsst.multi INFO: cancelling 17057295 on provider astro 2024-03-15 04:44:52 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro 2024-03-15 04:44:52 proc_lsst.multi:201 [INFO] cancelling 17057308 on provider astro proc_lsst.multi INFO: cancelling 17057308 on provider astro 2024-03-15 04:44:52 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro 2024-03-15 04:44:52 proc_lsst.multi:201 [INFO] cancelling 17057291 on provider astro proc_lsst.multi INFO: cancelling 17057291 on provider astro 2024-03-15 04:44:53 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro 2024-03-15 04:44:53 proc_lsst.multi:201 [INFO] cancelling 17057316 on provider astro proc_lsst.multi INFO: cancelling 17057316 on provider astro 2024-03-15 04:44:53 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint 2024-03-15 04:44:53 proc_lsst.multi:201 [INFO] cancelling 17057375 on provider checkpoint proc_lsst.multi INFO: cancelling 17057375 on provider checkpoint 2024-03-15 04:44:53 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k 2024-03-15 04:44:53 proc_lsst.multi:201 [INFO] cancelling 17057320 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17057320 on provider escience_rtx6k 2024-03-15 04:44:53 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint 2024-03-15 04:44:53 proc_lsst.multi:201 [INFO] cancelling 17057377 on provider checkpoint proc_lsst.multi INFO: cancelling 17057377 on provider checkpoint 2024-03-15 04:44:53 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 2024-03-15 04:44:53 proc_lsst.multi:201 [INFO] cancelling 17057331 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057331 on provider escience_a40 2024-03-15 04:44:54 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 2024-03-15 04:44:54 proc_lsst.multi:201 [INFO] cancelling 17057336 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057336 on provider escience_a40 2024-03-15 04:44:54 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro 2024-03-15 04:44:54 proc_lsst.multi:201 [INFO] cancelling 17057293 on provider astro proc_lsst.multi INFO: cancelling 17057293 on provider astro 2024-03-15 04:44:54 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 2024-03-15 04:44:54 proc_lsst.multi:201 [INFO] cancelling 17057333 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057333 on provider escience_a40 2024-03-15 04:44:54 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 2024-03-15 04:44:54 proc_lsst.multi:201 [INFO] cancelling 17057326 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057326 on provider escience_a40 2024-03-15 04:44:54 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint 2024-03-15 04:44:54 proc_lsst.multi:201 [INFO] cancelling 17057345 on provider checkpoint proc_lsst.multi INFO: cancelling 17057345 on provider checkpoint 2024-03-15 04:44:55 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro 2024-03-15 04:44:55 proc_lsst.multi:201 [INFO] cancelling 17057292 on provider astro proc_lsst.multi INFO: cancelling 17057292 on provider astro 2024-03-15 04:44:55 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint 2024-03-15 04:44:55 proc_lsst.multi:201 [INFO] cancelling 17057372 on provider checkpoint proc_lsst.multi INFO: cancelling 17057372 on provider checkpoint 2024-03-15 04:44:55 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro 2024-03-15 04:44:55 proc_lsst.multi:201 [INFO] cancelling 17057305 on provider astro proc_lsst.multi INFO: cancelling 17057305 on provider astro 2024-03-15 04:44:55 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint 2024-03-15 04:44:55 proc_lsst.multi:201 [INFO] cancelling 17057353 on provider checkpoint proc_lsst.multi INFO: cancelling 17057353 on provider checkpoint 2024-03-15 04:44:55 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 2024-03-15 04:44:55 proc_lsst.multi:201 [INFO] cancelling 17057338 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057338 on provider escience_a40 2024-03-15 04:44:55 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint 2024-03-15 04:44:55 proc_lsst.multi:201 [INFO] cancelling 17057349 on provider checkpoint proc_lsst.multi INFO: cancelling 17057349 on provider checkpoint 2024-03-15 04:44:56 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k 2024-03-15 04:44:56 proc_lsst.multi:201 [INFO] cancelling 17057319 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17057319 on provider escience_rtx6k 2024-03-15 04:44:56 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k 2024-03-15 04:44:56 proc_lsst.multi:201 [INFO] cancelling 17057318 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17057318 on provider escience_rtx6k 2024-03-15 04:44:56 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro 2024-03-15 04:44:56 proc_lsst.multi:201 [INFO] cancelling 17057298 on provider astro proc_lsst.multi INFO: cancelling 17057298 on provider astro 2024-03-15 04:44:56 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint 2024-03-15 04:44:56 proc_lsst.multi:201 [INFO] cancelling 17057350 on provider checkpoint proc_lsst.multi INFO: cancelling 17057350 on provider checkpoint 2024-03-15 04:44:56 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 2024-03-15 04:44:56 proc_lsst.multi:201 [INFO] cancelling 17057327 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057327 on provider escience_a40 2024-03-15 04:44:57 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint 2024-03-15 04:44:57 proc_lsst.multi:201 [INFO] cancelling 17057369 on provider checkpoint proc_lsst.multi INFO: cancelling 17057369 on provider checkpoint 2024-03-15 04:44:57 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro 2024-03-15 04:44:57 proc_lsst.multi:201 [INFO] cancelling 17057314 on provider astro proc_lsst.multi INFO: cancelling 17057314 on provider astro 2024-03-15 04:44:57 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 2024-03-15 04:44:57 proc_lsst.multi:201 [INFO] cancelling 17057325 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057325 on provider escience_a40 2024-03-15 04:44:57 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro 2024-03-15 04:44:57 proc_lsst.multi:201 [INFO] cancelling 17057296 on provider astro proc_lsst.multi INFO: cancelling 17057296 on provider astro 2024-03-15 04:44:57 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 2024-03-15 04:44:57 proc_lsst.multi:201 [INFO] cancelling 17057329 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057329 on provider escience_a40 2024-03-15 04:44:57 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro 2024-03-15 04:44:57 proc_lsst.multi:201 [INFO] cancelling 17057315 on provider astro proc_lsst.multi INFO: cancelling 17057315 on provider astro 2024-03-15 04:44:58 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 2024-03-15 04:44:58 proc_lsst.multi:201 [INFO] cancelling 17057330 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057330 on provider escience_a40 2024-03-15 04:44:58 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint 2024-03-15 04:44:58 proc_lsst.multi:201 [INFO] cancelling 17057374 on provider checkpoint proc_lsst.multi INFO: cancelling 17057374 on provider checkpoint 2024-03-15 04:44:58 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro 2024-03-15 04:44:58 proc_lsst.multi:201 [INFO] cancelling 17057313 on provider astro proc_lsst.multi INFO: cancelling 17057313 on provider astro 2024-03-15 04:44:58 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro 2024-03-15 04:44:58 proc_lsst.multi:201 [INFO] cancelling 17057299 on provider astro proc_lsst.multi INFO: cancelling 17057299 on provider astro 2024-03-15 04:44:59 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint 2024-03-15 04:44:59 proc_lsst.multi:201 [INFO] cancelling 17057368 on provider checkpoint proc_lsst.multi INFO: cancelling 17057368 on provider checkpoint 2024-03-15 04:44:59 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro 2024-03-15 04:44:59 proc_lsst.multi:201 [INFO] cancelling 17057300 on provider astro proc_lsst.multi INFO: cancelling 17057300 on provider astro 2024-03-15 04:44:59 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro 2024-03-15 04:44:59 proc_lsst.multi:201 [INFO] cancelling 17057287 on provider astro proc_lsst.multi INFO: cancelling 17057287 on provider astro 2024-03-15 04:44:59 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint 2024-03-15 04:44:59 proc_lsst.multi:201 [INFO] cancelling 17057346 on provider checkpoint proc_lsst.multi INFO: cancelling 17057346 on provider checkpoint 2024-03-15 04:44:59 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro 2024-03-15 04:44:59 proc_lsst.multi:201 [INFO] cancelling 17057311 on provider astro proc_lsst.multi INFO: cancelling 17057311 on provider astro 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057334 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057334 on provider escience_a40 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057328 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057328 on provider escience_a40 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057307 on provider astro proc_lsst.multi INFO: cancelling 17057307 on provider astro 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057302 on provider astro proc_lsst.multi INFO: cancelling 17057302 on provider astro 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057367 on provider checkpoint proc_lsst.multi INFO: cancelling 17057367 on provider checkpoint 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057297 on provider astro proc_lsst.multi INFO: cancelling 17057297 on provider astro 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057309 on provider astro proc_lsst.multi INFO: cancelling 17057309 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-15 04:45:00 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-15 04:45:00 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17057342', '17057343', '17057344', '17057345', '17057346', '17057347', '17057348', '17057349', '17057350', '17057351', '17057352', '17057353', '17057354', '17057355', '17057356', '17057357', '17057358', '17057359', '17057360', '17057361', '17057362', '17057363', '17057364', '17057365', '17057366', '17057367', '17057368', '17057369', '17057370', '17057371', '17057372', '17057373', '17057374', '17057375', '17057376', '17057377', '17057378', '17057287', '17057288', '17057289', '17057290', '17057291', '17057292', '17057293', '17057294', '17057295', '17057296', '17057297', '17057298', '17057299', '17057300', '17057301', '17057302', '17057303', '17057304', '17057305', '17057306', '17057307', '17057308', '17057309', '17057310', '17057311', '17057312', '17057313', '17057314', '17057315', '17057316', '17057322', '17057323', '17057324', '17057325', '17057326', '17057327', '17057328', '17057329', '17057330', '17057331', '17057332', '17057333', '17057334', '17057335', '17057336', '17057337', '17057338', '17057339', '17057340', '17057341', '17057317', '17057318', '17057319', '17057320', '17057321', '55323'] proc_lsst.multi INFO: new jobs since last cancel ['17057342', '17057343', '17057344', '17057345', '17057346', '17057347', '17057348', '17057349', '17057350', '17057351', '17057352', '17057353', '17057354', '17057355', '17057356', '17057357', '17057358', '17057359', '17057360', '17057361', '17057362', '17057363', '17057364', '17057365', '17057366', '17057367', '17057368', '17057369', '17057370', '17057371', '17057372', '17057373', '17057374', '17057375', '17057376', '17057377', '17057378', '17057287', '17057288', '17057289', '17057290', '17057291', '17057292', '17057293', '17057294', '17057295', '17057296', '17057297', '17057298', '17057299', '17057300', '17057301', '17057302', '17057303', '17057304', '17057305', '17057306', '17057307', '17057308', '17057309', '17057310', '17057311', '17057312', '17057313', '17057314', '17057315', '17057316', '17057322', '17057323', '17057324', '17057325', '17057326', '17057327', '17057328', '17057329', '17057330', '17057331', '17057332', '17057333', '17057334', '17057335', '17057336', '17057337', '17057338', '17057339', '17057340', '17057341', '17057317', '17057318', '17057319', '17057320', '17057321', '55323'] 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057342 in provider checkpoint proc_lsst.multi INFO: found job 17057342 in provider checkpoint 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057342 on provider checkpoint proc_lsst.multi INFO: cancelling 17057342 on provider checkpoint 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057343 in provider checkpoint proc_lsst.multi INFO: found job 17057343 in provider checkpoint 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057343 on provider checkpoint proc_lsst.multi INFO: cancelling 17057343 on provider checkpoint 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057344 in provider checkpoint proc_lsst.multi INFO: found job 17057344 in provider checkpoint 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057344 on provider checkpoint proc_lsst.multi INFO: cancelling 17057344 on provider checkpoint 2024-03-15 04:45:00 proc_lsst.multi:146 [INFO] found job 17057345 in provider checkpoint proc_lsst.multi INFO: found job 17057345 in provider checkpoint 2024-03-15 04:45:00 proc_lsst.multi:201 [INFO] cancelling 17057345 on provider checkpoint proc_lsst.multi INFO: cancelling 17057345 on provider checkpoint 2024-03-15 04:45:01 proc_lsst.multi:146 [INFO] found job 17057346 in provider checkpoint proc_lsst.multi INFO: found job 17057346 in provider checkpoint 2024-03-15 04:45:01 proc_lsst.multi:201 [INFO] cancelling 17057346 on provider checkpoint proc_lsst.multi INFO: cancelling 17057346 on provider checkpoint 2024-03-15 04:45:01 proc_lsst.multi:146 [INFO] found job 17057347 in provider checkpoint proc_lsst.multi INFO: found job 17057347 in provider checkpoint 2024-03-15 04:45:01 proc_lsst.multi:201 [INFO] cancelling 17057347 on provider checkpoint proc_lsst.multi INFO: cancelling 17057347 on provider checkpoint 2024-03-15 04:45:01 proc_lsst.multi:146 [INFO] found job 17057348 in provider checkpoint proc_lsst.multi INFO: found job 17057348 in provider checkpoint 2024-03-15 04:45:01 proc_lsst.multi:201 [INFO] cancelling 17057348 on provider checkpoint proc_lsst.multi INFO: cancelling 17057348 on provider checkpoint 2024-03-15 04:45:01 proc_lsst.multi:146 [INFO] found job 17057349 in provider checkpoint proc_lsst.multi INFO: found job 17057349 in provider checkpoint 2024-03-15 04:45:01 proc_lsst.multi:201 [INFO] cancelling 17057349 on provider checkpoint proc_lsst.multi INFO: cancelling 17057349 on provider checkpoint 2024-03-15 04:45:01 proc_lsst.multi:146 [INFO] found job 17057350 in provider checkpoint proc_lsst.multi INFO: found job 17057350 in provider checkpoint 2024-03-15 04:45:01 proc_lsst.multi:201 [INFO] cancelling 17057350 on provider checkpoint proc_lsst.multi INFO: cancelling 17057350 on provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:146 [INFO] found job 17057351 in provider checkpoint proc_lsst.multi INFO: found job 17057351 in provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:201 [INFO] cancelling 17057351 on provider checkpoint proc_lsst.multi INFO: cancelling 17057351 on provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:146 [INFO] found job 17057352 in provider checkpoint proc_lsst.multi INFO: found job 17057352 in provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:201 [INFO] cancelling 17057352 on provider checkpoint proc_lsst.multi INFO: cancelling 17057352 on provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:146 [INFO] found job 17057353 in provider checkpoint proc_lsst.multi INFO: found job 17057353 in provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:201 [INFO] cancelling 17057353 on provider checkpoint proc_lsst.multi INFO: cancelling 17057353 on provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:146 [INFO] found job 17057354 in provider checkpoint proc_lsst.multi INFO: found job 17057354 in provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:201 [INFO] cancelling 17057354 on provider checkpoint proc_lsst.multi INFO: cancelling 17057354 on provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:146 [INFO] found job 17057355 in provider checkpoint proc_lsst.multi INFO: found job 17057355 in provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:201 [INFO] cancelling 17057355 on provider checkpoint proc_lsst.multi INFO: cancelling 17057355 on provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:146 [INFO] found job 17057356 in provider checkpoint proc_lsst.multi INFO: found job 17057356 in provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:201 [INFO] cancelling 17057356 on provider checkpoint proc_lsst.multi INFO: cancelling 17057356 on provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:146 [INFO] found job 17057357 in provider checkpoint proc_lsst.multi INFO: found job 17057357 in provider checkpoint 2024-03-15 04:45:02 proc_lsst.multi:201 [INFO] cancelling 17057357 on provider checkpoint proc_lsst.multi INFO: cancelling 17057357 on provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:146 [INFO] found job 17057358 in provider checkpoint proc_lsst.multi INFO: found job 17057358 in provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:201 [INFO] cancelling 17057358 on provider checkpoint proc_lsst.multi INFO: cancelling 17057358 on provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:146 [INFO] found job 17057359 in provider checkpoint proc_lsst.multi INFO: found job 17057359 in provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:201 [INFO] cancelling 17057359 on provider checkpoint proc_lsst.multi INFO: cancelling 17057359 on provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:146 [INFO] found job 17057360 in provider checkpoint proc_lsst.multi INFO: found job 17057360 in provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:201 [INFO] cancelling 17057360 on provider checkpoint proc_lsst.multi INFO: cancelling 17057360 on provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:146 [INFO] found job 17057361 in provider checkpoint proc_lsst.multi INFO: found job 17057361 in provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:201 [INFO] cancelling 17057361 on provider checkpoint proc_lsst.multi INFO: cancelling 17057361 on provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:146 [INFO] found job 17057362 in provider checkpoint proc_lsst.multi INFO: found job 17057362 in provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:201 [INFO] cancelling 17057362 on provider checkpoint proc_lsst.multi INFO: cancelling 17057362 on provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:146 [INFO] found job 17057363 in provider checkpoint proc_lsst.multi INFO: found job 17057363 in provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:201 [INFO] cancelling 17057363 on provider checkpoint proc_lsst.multi INFO: cancelling 17057363 on provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:146 [INFO] found job 17057364 in provider checkpoint proc_lsst.multi INFO: found job 17057364 in provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:201 [INFO] cancelling 17057364 on provider checkpoint proc_lsst.multi INFO: cancelling 17057364 on provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:146 [INFO] found job 17057365 in provider checkpoint proc_lsst.multi INFO: found job 17057365 in provider checkpoint 2024-03-15 04:45:03 proc_lsst.multi:201 [INFO] cancelling 17057365 on provider checkpoint proc_lsst.multi INFO: cancelling 17057365 on provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:146 [INFO] found job 17057366 in provider checkpoint proc_lsst.multi INFO: found job 17057366 in provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:201 [INFO] cancelling 17057366 on provider checkpoint proc_lsst.multi INFO: cancelling 17057366 on provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:146 [INFO] found job 17057367 in provider checkpoint proc_lsst.multi INFO: found job 17057367 in provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:201 [INFO] cancelling 17057367 on provider checkpoint proc_lsst.multi INFO: cancelling 17057367 on provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:146 [INFO] found job 17057368 in provider checkpoint proc_lsst.multi INFO: found job 17057368 in provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:201 [INFO] cancelling 17057368 on provider checkpoint proc_lsst.multi INFO: cancelling 17057368 on provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:146 [INFO] found job 17057369 in provider checkpoint proc_lsst.multi INFO: found job 17057369 in provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:201 [INFO] cancelling 17057369 on provider checkpoint proc_lsst.multi INFO: cancelling 17057369 on provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:146 [INFO] found job 17057370 in provider checkpoint proc_lsst.multi INFO: found job 17057370 in provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:201 [INFO] cancelling 17057370 on provider checkpoint proc_lsst.multi INFO: cancelling 17057370 on provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:146 [INFO] found job 17057371 in provider checkpoint proc_lsst.multi INFO: found job 17057371 in provider checkpoint 2024-03-15 04:45:04 proc_lsst.multi:201 [INFO] cancelling 17057371 on provider checkpoint proc_lsst.multi INFO: cancelling 17057371 on provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:146 [INFO] found job 17057372 in provider checkpoint proc_lsst.multi INFO: found job 17057372 in provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:201 [INFO] cancelling 17057372 on provider checkpoint proc_lsst.multi INFO: cancelling 17057372 on provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:146 [INFO] found job 17057373 in provider checkpoint proc_lsst.multi INFO: found job 17057373 in provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:201 [INFO] cancelling 17057373 on provider checkpoint proc_lsst.multi INFO: cancelling 17057373 on provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:146 [INFO] found job 17057374 in provider checkpoint proc_lsst.multi INFO: found job 17057374 in provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:201 [INFO] cancelling 17057374 on provider checkpoint proc_lsst.multi INFO: cancelling 17057374 on provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:146 [INFO] found job 17057375 in provider checkpoint proc_lsst.multi INFO: found job 17057375 in provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:201 [INFO] cancelling 17057375 on provider checkpoint proc_lsst.multi INFO: cancelling 17057375 on provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:146 [INFO] found job 17057376 in provider checkpoint proc_lsst.multi INFO: found job 17057376 in provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:201 [INFO] cancelling 17057376 on provider checkpoint proc_lsst.multi INFO: cancelling 17057376 on provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:146 [INFO] found job 17057377 in provider checkpoint proc_lsst.multi INFO: found job 17057377 in provider checkpoint 2024-03-15 04:45:05 proc_lsst.multi:201 [INFO] cancelling 17057377 on provider checkpoint proc_lsst.multi INFO: cancelling 17057377 on provider checkpoint 2024-03-15 04:45:06 proc_lsst.multi:146 [INFO] found job 17057378 in provider checkpoint proc_lsst.multi INFO: found job 17057378 in provider checkpoint 2024-03-15 04:45:06 proc_lsst.multi:201 [INFO] cancelling 17057378 on provider checkpoint proc_lsst.multi INFO: cancelling 17057378 on provider checkpoint 2024-03-15 04:45:06 proc_lsst.multi:146 [INFO] found job 17057287 in provider astro proc_lsst.multi INFO: found job 17057287 in provider astro 2024-03-15 04:45:06 proc_lsst.multi:201 [INFO] cancelling 17057287 on provider astro proc_lsst.multi INFO: cancelling 17057287 on provider astro 2024-03-15 04:45:06 proc_lsst.multi:146 [INFO] found job 17057288 in provider astro proc_lsst.multi INFO: found job 17057288 in provider astro 2024-03-15 04:45:06 proc_lsst.multi:201 [INFO] cancelling 17057288 on provider astro proc_lsst.multi INFO: cancelling 17057288 on provider astro 2024-03-15 04:45:06 proc_lsst.multi:146 [INFO] found job 17057289 in provider astro proc_lsst.multi INFO: found job 17057289 in provider astro 2024-03-15 04:45:06 proc_lsst.multi:201 [INFO] cancelling 17057289 on provider astro proc_lsst.multi INFO: cancelling 17057289 on provider astro 2024-03-15 04:45:06 proc_lsst.multi:146 [INFO] found job 17057290 in provider astro proc_lsst.multi INFO: found job 17057290 in provider astro 2024-03-15 04:45:06 proc_lsst.multi:201 [INFO] cancelling 17057290 on provider astro proc_lsst.multi INFO: cancelling 17057290 on provider astro 2024-03-15 04:45:07 proc_lsst.multi:146 [INFO] found job 17057291 in provider astro proc_lsst.multi INFO: found job 17057291 in provider astro 2024-03-15 04:45:07 proc_lsst.multi:201 [INFO] cancelling 17057291 on provider astro proc_lsst.multi INFO: cancelling 17057291 on provider astro 2024-03-15 04:45:07 proc_lsst.multi:146 [INFO] found job 17057292 in provider astro proc_lsst.multi INFO: found job 17057292 in provider astro 2024-03-15 04:45:07 proc_lsst.multi:201 [INFO] cancelling 17057292 on provider astro proc_lsst.multi INFO: cancelling 17057292 on provider astro 2024-03-15 04:45:07 proc_lsst.multi:146 [INFO] found job 17057293 in provider astro proc_lsst.multi INFO: found job 17057293 in provider astro 2024-03-15 04:45:07 proc_lsst.multi:201 [INFO] cancelling 17057293 on provider astro proc_lsst.multi INFO: cancelling 17057293 on provider astro 2024-03-15 04:45:07 proc_lsst.multi:146 [INFO] found job 17057294 in provider astro proc_lsst.multi INFO: found job 17057294 in provider astro 2024-03-15 04:45:07 proc_lsst.multi:201 [INFO] cancelling 17057294 on provider astro proc_lsst.multi INFO: cancelling 17057294 on provider astro 2024-03-15 04:45:07 proc_lsst.multi:146 [INFO] found job 17057295 in provider astro proc_lsst.multi INFO: found job 17057295 in provider astro 2024-03-15 04:45:07 proc_lsst.multi:201 [INFO] cancelling 17057295 on provider astro proc_lsst.multi INFO: cancelling 17057295 on provider astro 2024-03-15 04:45:07 proc_lsst.multi:146 [INFO] found job 17057296 in provider astro proc_lsst.multi INFO: found job 17057296 in provider astro 2024-03-15 04:45:07 proc_lsst.multi:201 [INFO] cancelling 17057296 on provider astro proc_lsst.multi INFO: cancelling 17057296 on provider astro 2024-03-15 04:45:07 proc_lsst.multi:146 [INFO] found job 17057297 in provider astro proc_lsst.multi INFO: found job 17057297 in provider astro 2024-03-15 04:45:07 proc_lsst.multi:201 [INFO] cancelling 17057297 on provider astro proc_lsst.multi INFO: cancelling 17057297 on provider astro 2024-03-15 04:45:07 proc_lsst.multi:146 [INFO] found job 17057298 in provider astro proc_lsst.multi INFO: found job 17057298 in provider astro 2024-03-15 04:45:07 proc_lsst.multi:201 [INFO] cancelling 17057298 on provider astro proc_lsst.multi INFO: cancelling 17057298 on provider astro 2024-03-15 04:45:07 proc_lsst.multi:146 [INFO] found job 17057299 in provider astro proc_lsst.multi INFO: found job 17057299 in provider astro 2024-03-15 04:45:07 proc_lsst.multi:201 [INFO] cancelling 17057299 on provider astro proc_lsst.multi INFO: cancelling 17057299 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057300 in provider astro proc_lsst.multi INFO: found job 17057300 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057300 on provider astro proc_lsst.multi INFO: cancelling 17057300 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057301 in provider astro proc_lsst.multi INFO: found job 17057301 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057301 on provider astro proc_lsst.multi INFO: cancelling 17057301 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057302 in provider astro proc_lsst.multi INFO: found job 17057302 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057302 on provider astro proc_lsst.multi INFO: cancelling 17057302 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057303 in provider astro proc_lsst.multi INFO: found job 17057303 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057303 on provider astro proc_lsst.multi INFO: cancelling 17057303 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057304 in provider astro proc_lsst.multi INFO: found job 17057304 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057304 on provider astro proc_lsst.multi INFO: cancelling 17057304 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057305 in provider astro proc_lsst.multi INFO: found job 17057305 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057305 on provider astro proc_lsst.multi INFO: cancelling 17057305 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057306 in provider astro proc_lsst.multi INFO: found job 17057306 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057306 on provider astro proc_lsst.multi INFO: cancelling 17057306 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057307 in provider astro proc_lsst.multi INFO: found job 17057307 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057307 on provider astro proc_lsst.multi INFO: cancelling 17057307 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057308 in provider astro proc_lsst.multi INFO: found job 17057308 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057308 on provider astro proc_lsst.multi INFO: cancelling 17057308 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057309 in provider astro proc_lsst.multi INFO: found job 17057309 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057309 on provider astro proc_lsst.multi INFO: cancelling 17057309 on provider astro 2024-03-15 04:45:08 proc_lsst.multi:146 [INFO] found job 17057310 in provider astro proc_lsst.multi INFO: found job 17057310 in provider astro 2024-03-15 04:45:08 proc_lsst.multi:201 [INFO] cancelling 17057310 on provider astro proc_lsst.multi INFO: cancelling 17057310 on provider astro 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057311 in provider astro proc_lsst.multi INFO: found job 17057311 in provider astro 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057311 on provider astro proc_lsst.multi INFO: cancelling 17057311 on provider astro 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057312 in provider astro proc_lsst.multi INFO: found job 17057312 in provider astro 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057312 on provider astro proc_lsst.multi INFO: cancelling 17057312 on provider astro 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057313 in provider astro proc_lsst.multi INFO: found job 17057313 in provider astro 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057313 on provider astro proc_lsst.multi INFO: cancelling 17057313 on provider astro 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057314 in provider astro proc_lsst.multi INFO: found job 17057314 in provider astro 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057314 on provider astro proc_lsst.multi INFO: cancelling 17057314 on provider astro 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057315 in provider astro proc_lsst.multi INFO: found job 17057315 in provider astro 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057315 on provider astro proc_lsst.multi INFO: cancelling 17057315 on provider astro 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057316 in provider astro proc_lsst.multi INFO: found job 17057316 in provider astro 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057316 on provider astro proc_lsst.multi INFO: cancelling 17057316 on provider astro 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057322 in provider escience_a40 proc_lsst.multi INFO: found job 17057322 in provider escience_a40 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057322 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057322 on provider escience_a40 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057323 in provider escience_a40 proc_lsst.multi INFO: found job 17057323 in provider escience_a40 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057323 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057323 on provider escience_a40 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057324 in provider escience_a40 proc_lsst.multi INFO: found job 17057324 in provider escience_a40 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057324 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057324 on provider escience_a40 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057325 in provider escience_a40 proc_lsst.multi INFO: found job 17057325 in provider escience_a40 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057325 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057325 on provider escience_a40 2024-03-15 04:45:09 proc_lsst.multi:146 [INFO] found job 17057326 in provider escience_a40 proc_lsst.multi INFO: found job 17057326 in provider escience_a40 2024-03-15 04:45:09 proc_lsst.multi:201 [INFO] cancelling 17057326 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057326 on provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:146 [INFO] found job 17057327 in provider escience_a40 proc_lsst.multi INFO: found job 17057327 in provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:201 [INFO] cancelling 17057327 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057327 on provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:146 [INFO] found job 17057328 in provider escience_a40 proc_lsst.multi INFO: found job 17057328 in provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:201 [INFO] cancelling 17057328 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057328 on provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:146 [INFO] found job 17057329 in provider escience_a40 proc_lsst.multi INFO: found job 17057329 in provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:201 [INFO] cancelling 17057329 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057329 on provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:146 [INFO] found job 17057330 in provider escience_a40 proc_lsst.multi INFO: found job 17057330 in provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:201 [INFO] cancelling 17057330 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057330 on provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:146 [INFO] found job 17057331 in provider escience_a40 proc_lsst.multi INFO: found job 17057331 in provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:201 [INFO] cancelling 17057331 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057331 on provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:146 [INFO] found job 17057332 in provider escience_a40 proc_lsst.multi INFO: found job 17057332 in provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:201 [INFO] cancelling 17057332 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057332 on provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:146 [INFO] found job 17057333 in provider escience_a40 proc_lsst.multi INFO: found job 17057333 in provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:201 [INFO] cancelling 17057333 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057333 on provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:146 [INFO] found job 17057334 in provider escience_a40 proc_lsst.multi INFO: found job 17057334 in provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:201 [INFO] cancelling 17057334 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057334 on provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:146 [INFO] found job 17057335 in provider escience_a40 proc_lsst.multi INFO: found job 17057335 in provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:201 [INFO] cancelling 17057335 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057335 on provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:146 [INFO] found job 17057336 in provider escience_a40 proc_lsst.multi INFO: found job 17057336 in provider escience_a40 2024-03-15 04:45:10 proc_lsst.multi:201 [INFO] cancelling 17057336 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057336 on provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 17057337 in provider escience_a40 proc_lsst.multi INFO: found job 17057337 in provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 17057337 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057337 on provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 17057338 in provider escience_a40 proc_lsst.multi INFO: found job 17057338 in provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 17057338 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057338 on provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 17057339 in provider escience_a40 proc_lsst.multi INFO: found job 17057339 in provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 17057339 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057339 on provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 17057340 in provider escience_a40 proc_lsst.multi INFO: found job 17057340 in provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 17057340 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057340 on provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 17057341 in provider escience_a40 proc_lsst.multi INFO: found job 17057341 in provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 17057341 on provider escience_a40 proc_lsst.multi INFO: cancelling 17057341 on provider escience_a40 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 17057317 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057317 in provider escience_rtx6k 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 17057317 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17057317 on provider escience_rtx6k 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 17057318 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057318 in provider escience_rtx6k 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 17057318 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17057318 on provider escience_rtx6k 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 17057319 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057319 in provider escience_rtx6k 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 17057319 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17057319 on provider escience_rtx6k 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 17057320 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057320 in provider escience_rtx6k 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 17057320 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17057320 on provider escience_rtx6k 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 17057321 in provider escience_rtx6k proc_lsst.multi INFO: found job 17057321 in provider escience_rtx6k 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 17057321 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17057321 on provider escience_rtx6k 2024-03-15 04:45:11 proc_lsst.multi:146 [INFO] found job 55323 in provider local proc_lsst.multi INFO: found job 55323 in provider local 2024-03-15 04:45:11 proc_lsst.multi:201 [INFO] cancelling 55323 on provider local proc_lsst.multi INFO: cancelling 55323 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 55323 parsl.providers.local.local WARNING: Failed to kill PID: 55323 and child processes on local 2024-03-15 04:45:12 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 1957.5831 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210507_A0j_science#step1_20240315T041214Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 1957.5897 seconds; current memory usage: 0.394 Gibyte, delta: 0.012 Gibyte, peak delta: 0.012 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 1975.7988 seconds; current memory usage: 0.394 Gibyte, delta: 0.258 Gibyte, peak delta: 0.258 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.395 Gibyte (main), 0.395 Gibyte (largest child process) Run Id: None Run Name: DEEP_20210507_A0j_science#step1_20240315T041214Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 32m58.217s user 6m49.645s sys 4m32.591s